[RFC PATCH 0/3] A drm_plane API to support HDR planes

Tue May 18 23:00:17 UTC 2021

On 2021-05-18 16:19, Harry Wentland wrote:
> On 2021-05-18 3:56 a.m., Pekka Paalanen wrote:
>> On Mon, 17 May 2021 15:39:03 -0400
>> Vitaly Prosyak <vitaly.prosyak at amd.com> wrote:
>> 
>>> On 2021-05-17 12:48 p.m., Sebastian Wick wrote:
>>>> On 2021-05-17 10:57, Pekka Paalanen wrote:
>>>>> On Fri, 14 May 2021 17:05:11 -0400
>>>>> Harry Wentland <harry.wentland at amd.com> wrote:
>>>>> 
>>>>>> On 2021-04-27 10:50 a.m., Pekka Paalanen wrote:
>>>>>>> On Mon, 26 Apr 2021 13:38:49 -0400
>>>>>>> Harry Wentland <harry.wentland at amd.com> wrote:
>>>>> 
>>>>> ...
>>>>> 
>>>>>>>> ## Mastering Luminances
>>>>>>>> 
>>>>>>>> Now we are able to use the PQ 2084 EOTF to define the luminance 
>>>>>>>> of
>>>>>>>> pixels in absolute terms. Unfortunately we're again presented 
>>>>>>>> with
>>>>>>>> physical limitations of the display technologies on the market
>>>>>> today.
>>>>>>>> Here are a few examples of luminance ranges of displays.
>>>>>>>> 
>>>>>>>> | Display                  | Luminance range in nits |
>>>>>>>> | ------------------------ | ----------------------- |
>>>>>>>> | Typical PC display       | 0.3 - 200 |
>>>>>>>> | Excellent LCD HDTV       | 0.3 - 400 |
>>>>>>>> | HDR LCD w/ local dimming | 0.05 - 1,500 |
>>>>>>>> 
>>>>>>>> Since no display can currently show the full 0.0005 to 10,000 
>>>>>>>> nits
>>>>>>>> luminance range the display will need to tonemap the HDR 
>>>>>>>> content,
>>>>>> i.e
>>>>>>>> to fit the content within a display's capabilities. To assist 
>>>>>>>> with
>>>>>>>> tonemapping HDR content is usually accompanied with a metadata 
>>>>>>>> that
>>>>>>>> describes (among other things) the minimum and maximum mastering
>>>>>>>> luminance, i.e. the maximum and minimum luminance of the display
>>>>>> that
>>>>>>>> was used to master the HDR content.
>>>>>>>> 
>>>>>>>> The HDR metadata is currently defined on the drm_connector via 
>>>>>>>> the
>>>>>>>> hdr_output_metadata blob property.
>>>>>>>> 
>>>>>>>> It might be useful to define per-plane hdr metadata, as 
>>>>>>>> different
>>>>>>>> planes might have been mastered differently.
>>>>>>> 
>>>>>>> I don't think this would directly help with the dynamic range
>>>>>> blending
>>>>>>> problem. You still need to establish the mapping between the 
>>>>>>> optical
>>>>>>> values from two different EOTFs and dynamic ranges. Or can you 
>>>>>>> know
>>>>>>> which optical values match the mastering display maximum and 
>>>>>>> minimum
>>>>>>> luminances for not-PQ?
>>>>>>> 
>>>>>> 
>>>>>> My understanding of this is probably best illustrated by this 
>>>>>> example:
>>>>>> 
>>>>>> Assume HDR was mastered on a display with a maximum white level of 
>>>>>> 500
>>>>>> nits and played back on a display that supports a max white level 
>>>>>> of
>>>>>> 400
>>>>>> nits. If you know the mastering white level of 500 you know that
>>>>>> this is
>>>>>> the maximum value you need to compress down to 400 nits, allowing
>>>>>> you to
>>>>>> use the full extent of the 400 nits panel.
>>>>> 
>>>>> Right, but in the kernel, where do you get these nits values from?
>>>>> 
>>>>> hdr_output_metadata blob is infoframe data to the monitor. I think 
>>>>> this
>>>>> should be independent of the metadata used for color 
>>>>> transformations in
>>>>> the display pipeline before the monitor.
>>>>> 
>>>>> EDID may tell us the monitor HDR metadata, but again what is used 
>>>>> in
>>>>> the color transformations should be independent, because EDIDs lie,
>>>>> lighting environments change, and users have different preferences.
>>>>> 
>>>>> What about black levels?
>>>>> 
>>>>> Do you want to do black level adjustment?
>>>>> 
>>>>> How exactly should the compression work?
>>>>> 
>>>>> Where do you map the mid-tones?
>>>>> 
>>>>> What if the end user wants something different?
>>>> 
>>>> I suspect that this is not about tone mapping at all. The use cases
>>>> listed always have the display in PQ mode and just assume that no
>>>> content exceeds the PQ limitations. Then you can simply bring all
>>>> content to the color space with a matrix multiplication and then map 
>>>> the
>>>> linear light content somewhere into the PQ range. Tone mapping is
>>>> performed in the display only.
>> 
>> The use cases do use the word "desktop" though. Harry, could you 
>> expand
>> on this, are you seeking a design that is good for generic desktop
>> compositors too, or one that is more tailored to "embedded" video
>> player systems taking the most advantage of (potentially
>> fixed-function) hardware?
>> 
> 
> The goal is to enable this on a generic desktop, such as generic 
> Wayland
> implementations or ChromeOS. We're not looking for a custom solution 
> for
> some embedded systems, though the solution we end up with should 
> obviously
> not prevent an implementation on embedded video players.
> 
>> What matrix would one choose? Which render intent would it
>> correspond to?
>> 
>> If you need to adapt different dynamic ranges into the blending 
>> dynamic
>> range, would a simple linear transformation really be enough?
>> 
>>>> From a generic wayland compositor point of view this is 
>>>> uninteresting.
>>>> 
>>> It a compositor's decision to provide or not the metadata property to
>>> the kernel. The metadata can be available from one or multiple 
>>> clients
>>> or most likely not available at all.
>>> 
>>> Compositors may put a display in HDR10 ( when PQ 2084 INV EOTF and TM
>>> occurs in display ) or NATIVE mode and do not attach any metadata to 
>>> the
>>> connector and do TM in compositor.
>>> 
>>> It is all about user preference or compositor design, or a 
>>> combination
>>> of both options.
>> 
>> Indeed. The thing here is that you cannot just add KMS UAPI, you also
>> need to have the FOSS userspace to go with it. So you need to have 
>> your
>> audience defined, userspace patches written and reviewed and agreed
>> to be a good idea. I'm afraid this particular UAPI design would be
>> difficult to justify with Weston. Maybe Kodi is a better audience?
>> 
> 
> I'm not sure designing a UAPI for Kodi that's not going to work for
> Wayland-compositors is the right thing. From a KMS driver maintainer
> standpoint I don't want an API for each userspace.
> 
> The idea here is to do design and discussion in public so we can 
> eventually
> arrive at a UAPI for HDR and CM that works for Wayland and by extension
> for every other userspace.

Eventually we want to be able to drive displays in PQ mode in weston (I
think?) where the TF property could be used in some cases. So that
property seems good to me. The SDR boost property then might also be
useful but it depends on the details of the formula/mechanism that you
can come up with.

But I want to stress again that we're going to drive all display in
their native color space and dynamic range if possible where none of
those properties discussed are useful and without some kind of 3D LUT in
the plane's pixel pipeline we won't be able to make use of them at all.

The Kodi folks can probably give you better feedback and an actual
implementation in reasonable time because they want to drive the display
in PQ mode.

>> But then again, one also needs to consider whether it is enough for a
>> new UAPI to satisfy only part of the possible audience and then need
>> yet another new UAPI to satisfy the rest. Adding new UAPI requires
>> defining the interactions with all existing UAPI as well.
>> 
>> Maybe we do need several different UAPIs for the "same" things if the
>> hardware designs are too different to cater with just one.
>> 
> 
> I feel we should have a section in the RFC that sketches how different 
> HW
> deals with this currently. It would be good if we can arrive at a UAPI 
> that
> captures at least the common functionality of various HW.
> 
> Harry
> 
>> 
>> Thanks,
>> pq
>> 