[RFC 0/5] Introduce drm sharpening property

Thu Mar 28 10:04:56 UTC 2024

On Wed, 27 Mar 2024 13:29:16 +0200
Pekka Paalanen <pekka.paalanen at haloniitty.fi> wrote:

> On Wed, 27 Mar 2024 07:11:48 +0000
> "Garg, Nemesa" <nemesa.garg at intel.com> wrote:
> 
> > > -----Original Message-----
> > > From: Pekka Paalanen <pekka.paalanen at haloniitty.fi>
> > > Sent: Wednesday, March 13, 2024 3:07 PM
> > > To: Garg, Nemesa <nemesa.garg at intel.com>
> > > Cc: Simon Ser <contact at emersion.fr>; intel-gfx at lists.freedesktop.org; dri-
> > > devel at lists.freedesktop.org; G M, Adarsh <adarsh.g.m at intel.com>
> > > Subject: Re: [RFC 0/5] Introduce drm sharpening property
> > > 
> > > On Tue, 12 Mar 2024 16:26:00 +0200
> > > Pekka Paalanen <pekka.paalanen at haloniitty.fi> wrote:
> > >     
> > > > On Tue, 12 Mar 2024 08:30:34 +0000
> > > > "Garg, Nemesa" <nemesa.garg at intel.com> wrote:
> > > >    
> > > > > This  KMS property is not implementing any formula    
> > > >
> > > > Sure it is. Maybe Intel just does not want to tell what the algorithm
> > > > is, or maybe it's even patented.
> > > >    
> > > > > and the values
> > > > > that are being used are based on empirical analysis and certain
> > > > > experiments done on the hardware. These values are fixed and is not
> > > > > expected to change and this can change from vendor to vendor. The
> > > > > client can choose any sharpness value on the scale and on the basis
> > > > > of it the sharpness will be set. The sharpness effect can be changed
> > > > > from content to content and from display to display so user needs to
> > > > > adjust the optimum intensity value so as to get good experience on
> > > > > the screen.
> > > > >    
> > > >
> > > > IOW, it's an opaque box operation, and there is no way to reproduce
> > > > its results without the specific Intel hardware. Definitely no way to
> > > > reproduce its results in free open source software alone.
> > > >
> > > > Such opaque box operations can only occur after KMS blending, at the
> > > > CRTC or later stage. They cannot appear before blending, not in the
> > > > new KMS color pipeline design at least. The reason is that the modern
> > > > way to use KMS planes is opportunistic composition off-loading.
> > > > Opportunistic means that userspace decides from time to time whether
> > > > it composes the final picture using KMS or some other rendering method
> > > > (usually GPU and shaders). Since userspace will arbitrarily switch
> > > > between KMS and render composition, both must result in the exact same
> > > > image, or end users will observe unwanted flicker.
> > > >
> > > > Such opaque box operations are fine after blending, because there they
> > > > can be configured once and remain on forever. No switching, no flicker.    
> > > 
> > > If you want to see how sharpness property would apply in Wayland design, it
> > > would be in step 5 "Adjust (settings UI)" of
> > > https://gitlab.freedesktop.org/pq/color-and-hdr/-/blob/main/doc/color-
> > > management-model.md#compositor-color-management-model
> > > 
> > > To relate that diagram to KMS color processing, you can identify step 3 "Compose"
> > > as the KMS blending step. Everything before step 3 happens in KMS plane color
> > > processing, and steps 4-5 happen in KMS CRTC color processing.
> > > 
> > > Sharpening would essentially be a "compositor color effect", it just happens to be
> > > implementable only by specific Intel hardware.
> > > 
> > > If a color effect is dynamic or content-dependant, it will preclude colorimetric
> > > monitor calibration.
> > > 
> > > 
> > > Thanks,
> > > pq
> > > 
> > >     
> > > > Where does "sharpeness" operation occur in the Intel color processing
> > > > chain? Is it before or after blending?
> > > >     
> > Thank you for detail explanation and link.
> > Sharpness operation occur post blending in CRTC ie on the final 
> > composed output after blending . Yes Pekka you are right as per the 
> > diagram it is done at step 5  "Adjust (settings UI)").  I  will also document this thing 
> > along with documentation change.
> >   
> > > > What kind of transfer characteristics does it expect from the image,
> > > > and can those be realized with KMS CRTC properties if KMS is
> > > > configured such that the blending happens using some other characteristics    
> > > (e.g.    
> > > > blending in optical space)?
> > > >    
> > The filter values are not dependent/calculated on the inputs of 
> >  image but depending on the blending space and other inputs the 
> > blended output gets changed and the sharpness is applied post 
> > blending so according to the content user needs to adjust the 
> > strength value to get the better visual effect. So tuning of sharpness strength 
> > may be needed by user based on  the input contents and blending policy
> > to get the desired experience.
> >   
> > > > What about SDR vs. HDR imagery?
> > > >    
> > The interface can be used for both HDR and SDR. The effect is more prominent for SDR use cases.
> > For HDR filter values and tap value may change.  
> 
> Who will be providing these values?
> 
> The kernel driver cannot know if it is dealing with SDR or HDR or which
> transfer function is in effect at that point of the post-blending color
> pipeline.
> 
> If the UAPI is one "strength" value, then how can it work?
> 
> Maybe the UAPI needs more controls, if not providing all "filter and
> tap" values directly. Maybe all the filter and tap values should be
> provided by userspace?

Actually, is the hardware just doing a convolution with a filter
defined by the driver?

Convolution algorithm (it is a formula!) is pretty standard stuff I
believe. If the hardware is actually doing convolution, then the driver
really should be exposing the convolution operation. Then people can
choose to use it for sharpening with the Intel developed kernels, or
for custom effects with custom kernels. Everyone would win. Convolution
is also something that other hardware vendors could implement.

A convolution filter would fit very well in the new KMS color pipeline
design for post-compositing operations, too.

Is the sharpening element doing something similar to the unsharp
masking?

I suppose users might want different strength based on what kind of
content is the majority on the screen. That makes it something that a
Wayland compositor would adjust automatically based on Wayland content
type (similar to HDMI content type), for example.

Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/intel-gfx/attachments/20240328/5668c492/attachment-0001.sig>