## Introduction
We are looking to enable HDR support for a couple of single-plane and multi-plane scenarios. To do this effectively we recommend new interfaces to drm_plane. Below I'll give a bit of background on HDR and why we propose these interfaces.
## Defining a pixel's luminance
Currently the luminance space of pixels in a framebuffer/plane presented to the display is not well defined. It's usually assumed to be in a 2.2 or 2.4 gamma space and has no mapping to an absolute luminance value but is interpreted in relative terms.
Luminance can be measured and described in absolute terms as candela per meter squared, or cd/m2, or nits. Even though a pixel value can be mapped to luminance in a linear fashion to do so without losing a lot of detail requires 16-bpc color depth. The reason for this is that human perception can distinguish roughly between a 0.5-1% luminance delta. A linear representation is suboptimal, wasting precision in the highlights and losing precision in the shadows.
A gamma curve is a decent approximation to a human's perception of luminance, but the PQ (perceptual quantizer) function [1] improves on it. It also defines the luminance values in absolute terms, with the highest value being 10,000 nits and the lowest 0.0005 nits.
Using a content that's defined in PQ space we can approximate the real world in a much better way.
Here are some examples of real-life objects and their approximate luminance values:
| Object | Luminance in nits | | ----------------- | ----------------- | | Sun | 1.6 million | | Fluorescent light | 10,000 | | Highlights | 1,000 - sunlight | | White Objects | 250 - 1,000 | | Typical objects | 1 - 250 | | Shadows | 0.01 - 1 | | Ultra Blacks | 0 - 0.0005 |
## Describing the luminance space
**We propose a new drm_plane property to describe the Eletro-Optical Transfer Function (EOTF) with which its framebuffer was composed.** Examples of EOTF are:
| EOTF | Description | | --------- |:------------------------------------------------------------------------- | | Gamma 2.2 | a simple 2.2 gamma | | sRGB | 2.4 gamma with small initial linear section | | PQ 2084 | SMPTE ST 2084; used for HDR video and allows for up to 10,000 nit support | | Linear | Linear relationship between pixel value and luminance value |
## Mastering Luminances
Now we are able to use the PQ 2084 EOTF to define the luminance of pixels in absolute terms. Unfortunately we're again presented with physical limitations of the display technologies on the market today. Here are a few examples of luminance ranges of displays.
| Display | Luminance range in nits | | ------------------------ | ----------------------- | | Typical PC display | 0.3 - 200 | | Excellent LCD HDTV | 0.3 - 400 | | HDR LCD w/ local dimming | 0.05 - 1,500 |
Since no display can currently show the full 0.0005 to 10,000 nits luminance range the display will need to tonemap the HDR content, i.e to fit the content within a display's capabilities. To assist with tonemapping HDR content is usually accompanied with a metadata that describes (among other things) the minimum and maximum mastering luminance, i.e. the maximum and minimum luminance of the display that was used to master the HDR content.
The HDR metadata is currently defined on the drm_connector via the hdr_output_metadata blob property.
It might be useful to define per-plane hdr metadata, as different planes might have been mastered differently.
## SDR Luminance
Since SDR covers a smaller luminance range than HDR, an SDR plane might look dark when blended with HDR content. Since the max HDR luminance can be quite variable (200-1,500 nits on actual displays) it is best to make the SDR maximum luminance value configurable.
**We propose a drm_plane property to specfy the desired maximum luminance of the SDR plane in nits.** This allows us to map the SDR content predictably into HDR's absolute luminance space.
## Let There Be Color
So far we've only talked about luminance, ignoring colors altogether. Just like in the luminance space, traditionally the color space of display outputs has not been well defined. Similar to how an EOTF defines a mapping of pixel data to an absolute luminance value, the color space maps color information for each pixel onto the CIE 1931 chromaticity space. This can be thought of as a mapping to an absolute, real-life, color value.
A color space is defined by its primaries and white point. The primaries and white point are expressed as coordinates in the CIE 1931 color space. Think of the red primary as the reddest red that can be displayed within the color space. Same for green and blue.
Examples of color spaces are:
| Color Space | Description | | ----------- | ------------------------------------------ | | BT 601 | similar to BT 709 | | BT 709 | used by sRGB content; ~53% of BT 2020 | | DCI-P3 | used by most HDR displays; ~72% of BT 2020 | | BT 2020 | standard for most HDR content |
The color space is defined in DRM for YCbCr planes via the color_encoding property of the drm_plane.
**We propose to add definitions for the RGB variants of the BT color spaces.**
## Color Primaries and White Point
Just like displays can currently not represent the entire 0.0005 - 10,000 nits HDR range of the PQ 2084 EOTF, they are currently not capable of representing the entire BT.2020 color Gamut. For this reason video content will often specify the color primaries and white point used to master the video, in order to allow displays to be able to map the image as best as possible onto the display's gamut.
## Displays and Tonemapping
External displays are able to do their own tone and color mapping, based on the mastering luminance, color primaries, and white space defined in the HDR metadata.
Internal panels (which are currently few and far between) usually don't include the complex HW to do tone and color mapping on their own and will require the display driver to perform appropriate mapping.
## Pixel Formats
The pixel formats, such as ARGB8888, ARGB2101010, P010, or FP16 are unrelated to color space and EOTF definitions. HDR pixels can be formatted in different ways but in order to not lose precision HDR content requires at least 10 bpc precision. For this reason ARGB2101010, P010, and FP16 are the obvious candidates for HDR. ARGB2101010 and P010 have the advantage of requiring only half the bandwidth as FP16, while FP16 has the advantage of enough precision to operate in a linear space, i.e. without EOTF.
## Proposed use-cases
Although the userspace side of this work is still in the early stages it is clear that we will want to support the following two use-cases:
**One XRGB2101010 HDR Plane:** A single, composited plane of HDR content. The use-case is a video player on a desktop with the compositor owning the composition of SDR and HDR content. The content shall be PQ BT.2020 formatted. The drm_connector's hdr_output_metadata shall be set.
**One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc desktop plane, with a P010 HDR video plane underlayed. The HDR plane shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR boost value. The drm_connector's hdr_output_metadata shall be set.
**One XRGB8888 SDR Plane - HDR output:** In order to support a smooth transition we recommend an OS that supports HDR output to provide the hdr_output_metadata on the drm_connector to configure the output for HDR, even when the content is only SDR. This will allow for a smooth transition between SDR-only and HDR content. In this use-case the SDR max luminance value should be provided on the drm_plane.
In DCN we will de-PQ or de-Gamma all input in order to blend in linear space. For SDR content we will also apply any desired boost before blending. After blending we will then re-apply the PQ EOTF and do RGB to YCbCr conversion if needed.
## Summary of proposed interface changes
per drm_plane: - new RGB color space definitions, mirroring the existing YUV color space definitions - new transfer function property - new SDR maximum white level property
## References
[1] https://en.wikipedia.org/wiki/High-dynamic-range_video#Perceptual_Quantizer
## Further Reading
https://gitlab.freedesktop.org/swick/wayland-protocols/-/blob/color/unstable... http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP309.pdf https://app.spectracal.com/Documents/White%20Papers/HDR_Demystified.pdf
Bhawanpreet Lakha (3): drm/color: Add RGB Color encodings drm/color: Add Color transfer functions for HDR/SDR drm/color: Add sdr boost property
.../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 4 +- .../gpu/drm/arm/display/komeda/komeda_plane.c | 4 +- drivers/gpu/drm/arm/malidp_planes.c | 4 +- drivers/gpu/drm/armada/armada_overlay.c | 4 +- drivers/gpu/drm/drm_atomic_uapi.c | 8 ++ drivers/gpu/drm/drm_color_mgmt.c | 84 +++++++++++++++++-- drivers/gpu/drm/i915/display/intel_sprite.c | 4 +- .../drm/i915/display/skl_universal_plane.c | 4 +- drivers/gpu/drm/nouveau/dispnv04/overlay.c | 4 +- drivers/gpu/drm/omapdrm/omap_plane.c | 4 +- drivers/gpu/drm/sun4i/sun8i_vi_layer.c | 4 +- drivers/gpu/drm/tidss/tidss_plane.c | 6 +- include/drm/drm_color_mgmt.h | 25 +++++- include/drm/drm_plane.h | 30 +++++++ 14 files changed, 173 insertions(+), 16 deletions(-)
From: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com
Add the following color encodings - RGB versions for BT601, BT709, BT2020 - DCI-P3: Used for digital movies
Signed-off-by: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com Signed-off-by: Harry Wentland harry.wentland@amd.com --- drivers/gpu/drm/drm_color_mgmt.c | 4 ++++ include/drm/drm_color_mgmt.h | 4 ++++ 2 files changed, 8 insertions(+)
diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c index bb14f488c8f6..a183ebae2941 100644 --- a/drivers/gpu/drm/drm_color_mgmt.c +++ b/drivers/gpu/drm/drm_color_mgmt.c @@ -469,6 +469,10 @@ static const char * const color_encoding_name[] = { [DRM_COLOR_YCBCR_BT601] = "ITU-R BT.601 YCbCr", [DRM_COLOR_YCBCR_BT709] = "ITU-R BT.709 YCbCr", [DRM_COLOR_YCBCR_BT2020] = "ITU-R BT.2020 YCbCr", + [DRM_COLOR_RGB_BT601] = "ITU-R BT.601 RGB", + [DRM_COLOR_RGB_BT709] = "ITU-R BT.709 RGB", + [DRM_COLOR_RGB_BT2020] = "ITU-R BT.2020 RGB", + [DRM_COLOR_P3] = "DCI-P3", };
static const char * const color_range_name[] = { diff --git a/include/drm/drm_color_mgmt.h b/include/drm/drm_color_mgmt.h index 81c298488b0c..3043dd73480c 100644 --- a/include/drm/drm_color_mgmt.h +++ b/include/drm/drm_color_mgmt.h @@ -78,6 +78,10 @@ enum drm_color_encoding { DRM_COLOR_YCBCR_BT601, DRM_COLOR_YCBCR_BT709, DRM_COLOR_YCBCR_BT2020, + DRM_COLOR_RGB_BT601, + DRM_COLOR_RGB_BT709, + DRM_COLOR_RGB_BT2020, + DRM_COLOR_P3, DRM_COLOR_ENCODING_MAX, };
On Mon, Apr 26, 2021 at 01:38:50PM -0400, Harry Wentland wrote:
From: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com
Add the following color encodings
- RGB versions for BT601, BT709, BT2020
- DCI-P3: Used for digital movies
Signed-off-by: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com Signed-off-by: Harry Wentland harry.wentland@amd.com
drivers/gpu/drm/drm_color_mgmt.c | 4 ++++ include/drm/drm_color_mgmt.h | 4 ++++ 2 files changed, 8 insertions(+)
diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c index bb14f488c8f6..a183ebae2941 100644 --- a/drivers/gpu/drm/drm_color_mgmt.c +++ b/drivers/gpu/drm/drm_color_mgmt.c @@ -469,6 +469,10 @@ static const char * const color_encoding_name[] = { [DRM_COLOR_YCBCR_BT601] = "ITU-R BT.601 YCbCr", [DRM_COLOR_YCBCR_BT709] = "ITU-R BT.709 YCbCr", [DRM_COLOR_YCBCR_BT2020] = "ITU-R BT.2020 YCbCr",
- [DRM_COLOR_RGB_BT601] = "ITU-R BT.601 RGB",
- [DRM_COLOR_RGB_BT709] = "ITU-R BT.709 RGB",
- [DRM_COLOR_RGB_BT2020] = "ITU-R BT.2020 RGB",
- [DRM_COLOR_P3] = "DCI-P3",
These are a totally different thing than the YCbCr stuff. The YCbCr stuff just specifies the YCbCr<->RGB converison matrix, whereas these are I guess supposed to specify the primaries/whitepoint? But without specifying what we're converting *to* these mean absolutely nothing. Ie. I don't think they belong in this property.
The previous proposals around this topic have suggested a new property to specify a conversion matrix either explicitly, or via a separate enum (which would specify both the src and dst colorspaces). I've always argued the enum approach is needed anyway since not all hardware has a programmable matrix for this. But a fully programmable matrix could be nice for tone mapping purposes/etc, so we may want to make sure both are possible.
As for the transfer func, the proposals so far have mostly just been to expose a programmable degamma/gamma LUTs for each plane. But considering how poor the current gamma uapi is we've thrown around some ideas how to allow the kernel to properly expose the hw capabilities. This is one of those ideas: https://lists.freedesktop.org/archives/dri-devel/2019-April/212886.html I think that basic idea could be also be extended to allow fixed curves in case the hw doesn't have a fully programmable LUT. But dunno if that's relevant for your hw.
On 2021-04-26 2:07 p.m., Ville Syrjälä wrote:
On Mon, Apr 26, 2021 at 01:38:50PM -0400, Harry Wentland wrote:
From: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com
Add the following color encodings
- RGB versions for BT601, BT709, BT2020
- DCI-P3: Used for digital movies
Signed-off-by: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com Signed-off-by: Harry Wentland harry.wentland@amd.com
drivers/gpu/drm/drm_color_mgmt.c | 4 ++++ include/drm/drm_color_mgmt.h | 4 ++++ 2 files changed, 8 insertions(+)
diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c index bb14f488c8f6..a183ebae2941 100644 --- a/drivers/gpu/drm/drm_color_mgmt.c +++ b/drivers/gpu/drm/drm_color_mgmt.c @@ -469,6 +469,10 @@ static const char * const color_encoding_name[] = { [DRM_COLOR_YCBCR_BT601] = "ITU-R BT.601 YCbCr", [DRM_COLOR_YCBCR_BT709] = "ITU-R BT.709 YCbCr", [DRM_COLOR_YCBCR_BT2020] = "ITU-R BT.2020 YCbCr",
- [DRM_COLOR_RGB_BT601] = "ITU-R BT.601 RGB",
- [DRM_COLOR_RGB_BT709] = "ITU-R BT.709 RGB",
- [DRM_COLOR_RGB_BT2020] = "ITU-R BT.2020 RGB",
- [DRM_COLOR_P3] = "DCI-P3",
These are a totally different thing than the YCbCr stuff. The YCbCr stuff just specifies the YCbCr<->RGB converison matrix, whereas these are I guess supposed to specify the primaries/whitepoint? But without specifying what we're converting *to* these mean absolutely nothing. Ie. I don't think they belong in this property.
If this is the intention I don't see it documented.
I might have overlooked something but do we have a way to explicitly specify today what *to* format the YCbCr color encodings convert into? Would that be a combination of the output color encoding specified via colorspace_property and the color space encoded in the primaries and whitepoint of the hdr_output_metadata?
Fundamentally I don't see how the use of this property differs, whether you translate from YCbCr or from RGB. In either case you're converting from the defined input color space and pixel format to an output color space and pixel format.
The previous proposals around this topic have suggested a new property to specify a conversion matrix either explicitly, or via a separate enum (which would specify both the src and dst colorspaces). I've always argued the enum approach is needed anyway since not all hardware has a programmable matrix for this. But a fully programmable matrix could be nice for tone mapping purposes/etc, so we may want to make sure both are possible.
As for the transfer func, the proposals so far have mostly just been to expose a programmable degamma/gamma LUTs for each plane. But considering how poor the current gamma uapi is we've thrown around some ideas how to allow the kernel to properly expose the hw capabilities. This is one of those ideas: https://lists.freedesktop.org/archives/dri-devel/2019-April/212886.html%3E%3... I think that basic idea could be also be extended to allow fixed curves in case the hw doesn't have a fully programmable LUT. But dunno if that's relevant for your hw.
The problem with exposing gamma, whether per-plane or per-crtc, is that it is hard to define an API that works for all the HW out there. The capabilities for different HW differ a lot, not just between vendors but also between generations of a vendor's HW.
Another reason I'm proposing to define the color space (and gamma) of a plane is to make this explicit. Up until the color space and gamma of a plane or framebuffer are not well defined, which leads to drivers assuming the color space and gamma of a buffer (for blending and other purposes) and might lead to sub-optimal outcomes.
Harry
On Mon, Apr 26, 2021 at 02:56:26PM -0400, Harry Wentland wrote:
On 2021-04-26 2:07 p.m., Ville Syrjälä wrote:
On Mon, Apr 26, 2021 at 01:38:50PM -0400, Harry Wentland wrote:
From: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com
Add the following color encodings
- RGB versions for BT601, BT709, BT2020
- DCI-P3: Used for digital movies
Signed-off-by: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com Signed-off-by: Harry Wentland harry.wentland@amd.com
drivers/gpu/drm/drm_color_mgmt.c | 4 ++++ include/drm/drm_color_mgmt.h | 4 ++++ 2 files changed, 8 insertions(+)
diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c index bb14f488c8f6..a183ebae2941 100644 --- a/drivers/gpu/drm/drm_color_mgmt.c +++ b/drivers/gpu/drm/drm_color_mgmt.c @@ -469,6 +469,10 @@ static const char * const color_encoding_name[] = { [DRM_COLOR_YCBCR_BT601] = "ITU-R BT.601 YCbCr", [DRM_COLOR_YCBCR_BT709] = "ITU-R BT.709 YCbCr", [DRM_COLOR_YCBCR_BT2020] = "ITU-R BT.2020 YCbCr",
- [DRM_COLOR_RGB_BT601] = "ITU-R BT.601 RGB",
- [DRM_COLOR_RGB_BT709] = "ITU-R BT.709 RGB",
- [DRM_COLOR_RGB_BT2020] = "ITU-R BT.2020 RGB",
- [DRM_COLOR_P3] = "DCI-P3",
These are a totally different thing than the YCbCr stuff. The YCbCr stuff just specifies the YCbCr<->RGB converison matrix, whereas these are I guess supposed to specify the primaries/whitepoint? But without specifying what we're converting *to* these mean absolutely nothing. Ie. I don't think they belong in this property.
If this is the intention I don't see it documented.
I might have overlooked something but do we have a way to explicitly specify today what *to* format the YCbCr color encodings convert into?
These just specific which YCbCr<->RGB matrix to use as specificed in the relevant standards. The primaries/whitepoint/etc. don't change at all.
Would that be a combination of the output color encoding specified via colorspace_property and the color space encoded in the primaries and whitepoint of the hdr_output_metadata?
Those propertis only affect the infoframes. They don't apply any color processing to the data.
Fundamentally I don't see how the use of this property differs, whether you translate from YCbCr or from RGB. In either case you're converting from the defined input color space and pixel format to an output color space and pixel format.
The gamut does not change when you do YCbCr<->RGB conversion.
The previous proposals around this topic have suggested a new property to specify a conversion matrix either explicitly, or via a separate enum (which would specify both the src and dst colorspaces). I've always argued the enum approach is needed anyway since not all hardware has a programmable matrix for this. But a fully programmable matrix could be nice for tone mapping purposes/etc, so we may want to make sure both are possible.
As for the transfer func, the proposals so far have mostly just been to expose a programmable degamma/gamma LUTs for each plane. But considering how poor the current gamma uapi is we've thrown around some ideas how to allow the kernel to properly expose the hw capabilities. This is one of those ideas: https://lists.freedesktop.org/archives/dri-devel/2019-April/212886.html%3E%3... I think that basic idea could be also be extended to allow fixed curves in case the hw doesn't have a fully programmable LUT. But dunno if that's relevant for your hw.
The problem with exposing gamma, whether per-plane or per-crtc, is that it is hard to define an API that works for all the HW out there. The capabilities for different HW differ a lot, not just between vendors but also between generations of a vendor's HW.
Another reason I'm proposing to define the color space (and gamma) of a plane is to make this explicit. Up until the color space and gamma of a plane or framebuffer are not well defined, which leads to drivers assuming the color space and gamma of a buffer (for blending and other purposes) and might lead to sub-optimal outcomes.
The current state is that things just get passed through as is (apart from the crtc LUTs/CTM).
On Mon, 26 Apr 2021 22:08:55 +0300 Ville Syrjälä ville.syrjala@linux.intel.com wrote:
On Mon, Apr 26, 2021 at 02:56:26PM -0400, Harry Wentland wrote:
On 2021-04-26 2:07 p.m., Ville Syrjälä wrote:
On Mon, Apr 26, 2021 at 01:38:50PM -0400, Harry Wentland wrote:
From: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com
Add the following color encodings
- RGB versions for BT601, BT709, BT2020
- DCI-P3: Used for digital movies
Signed-off-by: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com Signed-off-by: Harry Wentland harry.wentland@amd.com
drivers/gpu/drm/drm_color_mgmt.c | 4 ++++ include/drm/drm_color_mgmt.h | 4 ++++ 2 files changed, 8 insertions(+)
diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c index bb14f488c8f6..a183ebae2941 100644 --- a/drivers/gpu/drm/drm_color_mgmt.c +++ b/drivers/gpu/drm/drm_color_mgmt.c @@ -469,6 +469,10 @@ static const char * const color_encoding_name[] = { [DRM_COLOR_YCBCR_BT601] = "ITU-R BT.601 YCbCr", [DRM_COLOR_YCBCR_BT709] = "ITU-R BT.709 YCbCr", [DRM_COLOR_YCBCR_BT2020] = "ITU-R BT.2020 YCbCr",
- [DRM_COLOR_RGB_BT601] = "ITU-R BT.601 RGB",
- [DRM_COLOR_RGB_BT709] = "ITU-R BT.709 RGB",
- [DRM_COLOR_RGB_BT2020] = "ITU-R BT.2020 RGB",
- [DRM_COLOR_P3] = "DCI-P3",
These are a totally different thing than the YCbCr stuff. The YCbCr stuff just specifies the YCbCr<->RGB converison matrix, whereas these are I guess supposed to specify the primaries/whitepoint? But without specifying what we're converting *to* these mean absolutely nothing. Ie. I don't think they belong in this property.
If this is the intention I don't see it documented.
I might have overlooked something but do we have a way to explicitly specify today what *to* format the YCbCr color encodings convert into?
These just specific which YCbCr<->RGB matrix to use as specificed in the relevant standards. The primaries/whitepoint/etc. don't change at all.
Ville is correct here.
Would that be a combination of the output color encoding specified via colorspace_property and the color space encoded in the primaries and whitepoint of the hdr_output_metadata?
Conversion between YCbCR and RGB is not a color space conversion in the sense of color spaces (chromaticity of primaries and white point). It is a color model conversion or a color encoding conversion more like.
A benefit of YCbCr is that you can use less bandwidth to transmit the same image and people won't realise that you lost anything: chroma sub-sampling. Sub-sampling with RGB wouldn't work that well. It's a lossy compression technique, but different standards use different compression algorithms (well, matrices) to balance what gets lost.
Those propertis only affect the infoframes. They don't apply any color processing to the data.
Indeed.
An example:
You start with YUV video you want to display. That means you have YCbCr data using color space X and EOTF Foo. When you convert that to RGB, the RGB data still has color space X and EOTF Foo. Then you use the infoframe to tell your monitor that the data is in color space X and using EOTF Foo.
At no point in that pipeline there is a color space transformation, until the data actually reaches the monitor which may do magic things to map color space X and EOTF Foo into what it can actually make as light.
Or as with the traditional way, you don't care what color space or EOTF your video uses or your monitor has. You just hope they are close enough to look good enough that people don't see anything wrong. Close your eyes and sing a happy song. With HDR and WCG, that totally breaks down.
Fundamentally I don't see how the use of this property differs, whether you translate from YCbCr or from RGB. In either case you're converting from the defined input color space and pixel format to an output color space and pixel format.
The gamut does not change when you do YCbCr<->RGB conversion.
Right. Neither does dynamic range.
The previous proposals around this topic have suggested a new property to specify a conversion matrix either explicitly, or via a separate enum (which would specify both the src and dst colorspaces). I've always argued the enum approach is needed anyway since not all hardware has a programmable matrix for this. But a fully programmable matrix could be nice for tone mapping purposes/etc, so we may want to make sure both are possible.
As for the transfer func, the proposals so far have mostly just been to expose a programmable degamma/gamma LUTs for each plane. But considering how poor the current gamma uapi is we've thrown around some ideas how to allow the kernel to properly expose the hw capabilities. This is one of those ideas: https://lists.freedesktop.org/archives/dri-devel/2019-April/212886.html%3E%3... I think that basic idea could be also be extended to allow fixed curves in case the hw doesn't have a fully programmable LUT. But dunno if that's relevant for your hw.
The problem with exposing gamma, whether per-plane or per-crtc, is that it is hard to define an API that works for all the HW out there. The capabilities for different HW differ a lot, not just between vendors but also between generations of a vendor's HW.
Another reason I'm proposing to define the color space (and gamma) of a plane is to make this explicit. Up until the color space and gamma of a plane or framebuffer are not well defined, which leads to drivers assuming the color space and gamma of a buffer (for blending and other purposes) and might lead to sub-optimal outcomes.
The current state is that things just get passed through as is (apart from the crtc LUTs/CTM).
Right. I would claim that the kernel does not even want to know about color spaces or EOTFs. Instead, the kernel should offer userspace ways to program the hardware to do the color *transformations* it wants to do. The color_encoding KMS property is already like this: it defines the conversion matrix, not what the input or output are.
Infoframes being sent to displays are a different thing. They just tell the monitor what kind of image data userspace has configured KMS to send it, but does not change what KMS actually does with pixels.
Also, please, let's talk about EOTF and EOTF^-1 instead of gamma when appropriate.
Electro-optical transfer function (EOTF) is very clear in what it means: it is the mapping from electrical values (the non-linear pixel values you are used to, good for storage and transmission) to optical values (values that are linear in light intensity, therefore good for things like blending and filtering).
Gamma is kind of the same, but when you use it in sentences it easily becomes ambiguous. Like if you have "gamma corrected pixels", what does that mean. Are they electrical values or optical values or maybe electrical values with a different EOTF. What EOTF.
However, in KMS "gamma LUT" is kind of standardised terminology, and it does not need to be an EOTF or inverse-EOTF. One can use a gamma LUT for EETF, mapping from one EOTF to another, e.g. from data with content EOTF to data with monitor EOTF.
Thanks, pq
On 2021-04-26 20:56, Harry Wentland wrote:
On 2021-04-26 2:07 p.m., Ville Syrjälä wrote:
On Mon, Apr 26, 2021 at 01:38:50PM -0400, Harry Wentland wrote:
From: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com
Add the following color encodings
- RGB versions for BT601, BT709, BT2020
- DCI-P3: Used for digital movies
Signed-off-by: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com Signed-off-by: Harry Wentland harry.wentland@amd.com
drivers/gpu/drm/drm_color_mgmt.c | 4 ++++ include/drm/drm_color_mgmt.h | 4 ++++ 2 files changed, 8 insertions(+)
diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c index bb14f488c8f6..a183ebae2941 100644 --- a/drivers/gpu/drm/drm_color_mgmt.c +++ b/drivers/gpu/drm/drm_color_mgmt.c @@ -469,6 +469,10 @@ static const char * const color_encoding_name[] = { [DRM_COLOR_YCBCR_BT601] = "ITU-R BT.601 YCbCr", [DRM_COLOR_YCBCR_BT709] = "ITU-R BT.709 YCbCr", [DRM_COLOR_YCBCR_BT2020] = "ITU-R BT.2020 YCbCr",
- [DRM_COLOR_RGB_BT601] = "ITU-R BT.601 RGB",
- [DRM_COLOR_RGB_BT709] = "ITU-R BT.709 RGB",
- [DRM_COLOR_RGB_BT2020] = "ITU-R BT.2020 RGB",
- [DRM_COLOR_P3] = "DCI-P3",
These are a totally different thing than the YCbCr stuff. The YCbCr stuff just specifies the YCbCr<->RGB converison matrix, whereas these are I guess supposed to specify the primaries/whitepoint? But without specifying what we're converting *to* these mean absolutely nothing. Ie. I don't think they belong in this property.
If this is the intention I don't see it documented.
I might have overlooked something but do we have a way to explicitly specify today what *to* format the YCbCr color encodings convert into? Would that be a combination of the output color encoding specified via colorspace_property and the color space encoded in the primaries and whitepoint of the hdr_output_metadata?
Fundamentally I don't see how the use of this property differs, whether you translate from YCbCr or from RGB. In either case you're converting from the defined input color space and pixel format to an output color space and pixel format.
The previous proposals around this topic have suggested a new property to specify a conversion matrix either explicitly, or via a separate enum (which would specify both the src and dst colorspaces). I've always argued the enum approach is needed anyway since not all hardware has a programmable matrix for this. But a fully programmable matrix could be nice for tone mapping purposes/etc, so we may want to make sure both are possible.
As for the transfer func, the proposals so far have mostly just been to expose a programmable degamma/gamma LUTs for each plane. But considering how poor the current gamma uapi is we've thrown around some ideas how to allow the kernel to properly expose the hw capabilities. This is one of those ideas: https://lists.freedesktop.org/archives/dri-devel/2019-April/212886.html%3E%3... I think that basic idea could be also be extended to allow fixed curves in case the hw doesn't have a fully programmable LUT. But dunno if that's relevant for your hw.
The problem with exposing gamma, whether per-plane or per-crtc, is that it is hard to define an API that works for all the HW out there. The capabilities for different HW differ a lot, not just between vendors but also between generations of a vendor's HW.
Introducing another API if hardware is sufficiently different doesn't seem like the worst idea. At least it sounds a lot more tractable than teaching the kernel about all the different use cases, opinions and nuances that arise from color management.
In the end generic user space must always be able to fall back to software so the worst case is that it won't be able to offload an operation if it doesn't know about a new API.
Another reason I'm proposing to define the color space (and gamma) of a plane is to make this explicit. Up until the color space and gamma of a plane or framebuffer are not well defined, which leads to drivers assuming the color space and gamma of a buffer (for blending and other purposes) and might lead to sub-optimal outcomes.
Blending only is "correct" with linear light so that property of the color space is important. However, why does the kernel have to be involved here? As long as user space knows that for correct blending the data must represent linear light and knows when in the pipeline blending happens it can make sure that the data at that point in the pipeline contains linear light.
What other purposes are there?
In general I agree with the others that user space only wants a pipeline of transformations where the mechanism, the order and ideally the precision is defined.
Harry
On 2021-04-30 8:53 p.m., Sebastian Wick wrote:
On 2021-04-26 20:56, Harry Wentland wrote:
On 2021-04-26 2:07 p.m., Ville Syrjälä wrote:
On Mon, Apr 26, 2021 at 01:38:50PM -0400, Harry Wentland wrote:
From: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com
Add the following color encodings
- RGB versions for BT601, BT709, BT2020
- DCI-P3: Used for digital movies
Signed-off-by: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com Signed-off-by: Harry Wentland harry.wentland@amd.com
drivers/gpu/drm/drm_color_mgmt.c | 4 ++++ include/drm/drm_color_mgmt.h | 4 ++++ 2 files changed, 8 insertions(+)
diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c index bb14f488c8f6..a183ebae2941 100644 --- a/drivers/gpu/drm/drm_color_mgmt.c +++ b/drivers/gpu/drm/drm_color_mgmt.c @@ -469,6 +469,10 @@ static const char * const color_encoding_name[] = { [DRM_COLOR_YCBCR_BT601] = "ITU-R BT.601 YCbCr", [DRM_COLOR_YCBCR_BT709] = "ITU-R BT.709 YCbCr", [DRM_COLOR_YCBCR_BT2020] = "ITU-R BT.2020 YCbCr", + [DRM_COLOR_RGB_BT601] = "ITU-R BT.601 RGB", + [DRM_COLOR_RGB_BT709] = "ITU-R BT.709 RGB", + [DRM_COLOR_RGB_BT2020] = "ITU-R BT.2020 RGB", + [DRM_COLOR_P3] = "DCI-P3",
These are a totally different thing than the YCbCr stuff. The YCbCr stuff just specifies the YCbCr<->RGB converison matrix, whereas these are I guess supposed to specify the primaries/whitepoint? But without specifying what we're converting *to* these mean absolutely nothing. Ie. I don't think they belong in this property.
If this is the intention I don't see it documented.
I might have overlooked something but do we have a way to explicitly specify today what *to* format the YCbCr color encodings convert into? Would that be a combination of the output color encoding specified via colorspace_property and the color space encoded in the primaries and whitepoint of the hdr_output_metadata?
Fundamentally I don't see how the use of this property differs, whether you translate from YCbCr or from RGB. In either case you're converting from the defined input color space and pixel format to an output color space and pixel format.
The previous proposals around this topic have suggested a new property to specify a conversion matrix either explicitly, or via a separate enum (which would specify both the src and dst colorspaces). I've always argued the enum approach is needed anyway since not all hardware has a programmable matrix for this. But a fully programmable matrix could be nice for tone mapping purposes/etc, so we may want to make sure both are possible.
As for the transfer func, the proposals so far have mostly just been to expose a programmable degamma/gamma LUTs for each plane. But considering how poor the current gamma uapi is we've thrown around some ideas how to allow the kernel to properly expose the hw capabilities. This is one of those ideas: https://lists.freedesktop.org/archives/dri-devel/2019-April/212886.html I think that basic idea could be also be extended to allow fixed curves in case the hw doesn't have a fully programmable LUT. But dunno if that's relevant for your hw.
The problem with exposing gamma, whether per-plane or per-crtc, is that it is hard to define an API that works for all the HW out there. The capabilities for different HW differ a lot, not just between vendors but also between generations of a vendor's HW.
Introducing another API if hardware is sufficiently different doesn't seem like the worst idea. At least it sounds a lot more tractable than teaching the kernel about all the different use cases, opinions and nuances that arise from color management.
In the end generic user space must always be able to fall back to software so the worst case is that it won't be able to offload an operation if it doesn't know about a new API.
Another reason I'm proposing to define the color space (and gamma) of a plane is to make this explicit. Up until the color space and gamma of a plane or framebuffer are not well defined, which leads to drivers assuming the color space and gamma of a buffer (for blending and other purposes) and might lead to sub-optimal outcomes.
Blending only is "correct" with linear light so that property of the color space is important. However, why does the kernel have to be involved here? As long as user space knows that for correct blending the data must represent linear light and knows when in the pipeline blending happens it can make sure that the data at that point in the pipeline contains linear light.
The only reason the kernel needs to be involved is to take full advantage of the available HW without requiring KMS clients to be aware of the difference in display HW.
Harry
What other purposes are there?
In general I agree with the others that user space only wants a pipeline of transformations where the mechanism, the order and ideally the precision is defined.
Harry
amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx%3E
On Fri, 14 May 2021 17:04:51 -0400 Harry Wentland harry.wentland@amd.com wrote:
On 2021-04-30 8:53 p.m., Sebastian Wick wrote:
On 2021-04-26 20:56, Harry Wentland wrote:
...
Another reason I'm proposing to define the color space (and gamma) of a plane is to make this explicit. Up until the color space and gamma of a plane or framebuffer are not well defined, which leads to drivers assuming the color space and gamma of a buffer (for blending and other purposes) and might lead to sub-optimal outcomes.
Blending only is "correct" with linear light so that property of the color space is important. However, why does the kernel have to be involved here? As long as user space knows that for correct blending the data must represent linear light and knows when in the pipeline blending happens it can make sure that the data at that point in the pipeline contains linear light.
The only reason the kernel needs to be involved is to take full advantage of the available HW without requiring KMS clients to be aware of the difference in display HW.
Can you explain in more tangible examples, why you think so, please?
Is it because hardware does not fit the KMS UAPI model of the abstract pixel pipeline?
Or is it because you have fixed-function hardware elements that you can only make use of when userspace uses an enum-based UAPI?
I would totally agree that the driver does not want to be analysing LUT entries to decipher if it could use a fixed-function element or not. It would introduce uncertainty in the UAPI. So fixed-function elements would need their own properties, but I don't know if that is feasible as generic UAPI or if it should be driver-specific (and so left unused by generic userspace).
Thanks, pq
On 2021-05-17 4:34 a.m., Pekka Paalanen wrote:
On Fri, 14 May 2021 17:04:51 -0400 Harry Wentland harry.wentland@amd.com wrote:
On 2021-04-30 8:53 p.m., Sebastian Wick wrote:
On 2021-04-26 20:56, Harry Wentland wrote:
...
Another reason I'm proposing to define the color space (and gamma) of a plane is to make this explicit. Up until the color space and gamma of a plane or framebuffer are not well defined, which leads to drivers assuming the color space and gamma of a buffer (for blending and other purposes) and might lead to sub-optimal outcomes.
Blending only is "correct" with linear light so that property of the color space is important. However, why does the kernel have to be involved here? As long as user space knows that for correct blending the data must represent linear light and knows when in the pipeline blending happens it can make sure that the data at that point in the pipeline contains linear light.
The only reason the kernel needs to be involved is to take full advantage of the available HW without requiring KMS clients to be aware of the difference in display HW.
Can you explain in more tangible examples, why you think so, please?
Is it because hardware does not fit the KMS UAPI model of the abstract pixel pipeline?
I'd wager no HW is designed to meet KMS UAPI, rather KMS UAPI is designed to abstract HW.
Or is it because you have fixed-function hardware elements that you can only make use of when userspace uses an enum-based UAPI?
One example is our degamma on our latest generation HW, where we have fixed-function "degamma" (rather de-EOTF):
https://gitlab.freedesktop.org/agd5f/linux/-/blob/amd-staging-drm-next/drive...
I would totally agree that the driver does not want to be analysing LUT entries to decipher if it could use a fixed-function element or not. It would introduce uncertainty in the UAPI. So fixed-function elements would need their own properties, but I don't know if that is feasible as generic UAPI or if it should be driver-specific (and so left unused by generic userspace).
For the CRTC LUTs we actually do a linearity check to program the HW into bypass when the LUT is linear since the KMS LUT definition doesn't map well onto the LUT definition used by our HW and leads to rounding errors and failing IGT kms_color tests (if I remember this correctly).
https://gitlab.freedesktop.org/agd5f/linux/-/blob/amd-staging-drm-next/drive...
Hence the suggestion to define pre-defined TFs right at a KMS level for usecases where we can assume the display will tonemap the content.
Harry
Thanks, pq
On Tue, 18 May 2021 10:32:48 -0400 Harry Wentland harry.wentland@amd.com wrote:
On 2021-05-17 4:34 a.m., Pekka Paalanen wrote:
On Fri, 14 May 2021 17:04:51 -0400 Harry Wentland harry.wentland@amd.com wrote:
On 2021-04-30 8:53 p.m., Sebastian Wick wrote:
On 2021-04-26 20:56, Harry Wentland wrote:
...
Another reason I'm proposing to define the color space (and gamma) of a plane is to make this explicit. Up until the color space and gamma of a plane or framebuffer are not well defined, which leads to drivers assuming the color space and gamma of a buffer (for blending and other purposes) and might lead to sub-optimal outcomes.
Blending only is "correct" with linear light so that property of the color space is important. However, why does the kernel have to be involved here? As long as user space knows that for correct blending the data must represent linear light and knows when in the pipeline blending happens it can make sure that the data at that point in the pipeline contains linear light.
The only reason the kernel needs to be involved is to take full advantage of the available HW without requiring KMS clients to be aware of the difference in display HW.
Can you explain in more tangible examples, why you think so, please?
Is it because hardware does not fit the KMS UAPI model of the abstract pixel pipeline?
I'd wager no HW is designed to meet KMS UAPI, rather KMS UAPI is designed to abstract HW.
Of course, but you are in big trouble in any case if there is a fundamental mismatch. You may have to declare that all existing KMS properties for this stuff will be mutually exclusive with your new properties, so that you can introduce a new generic abstract pipeline in KMS.
By mutually exclusive I mean that a driver must advertise only one or the other set of properties and never both. If you want to support userspace that doesn't understand the alternative set, maybe you also need a drm client cap to switch to the alternative set per-drm-client.
Or is it because you have fixed-function hardware elements that you can only make use of when userspace uses an enum-based UAPI?
One example is our degamma on our latest generation HW, where we have fixed-function "degamma" (rather de-EOTF):
https://gitlab.freedesktop.org/agd5f/linux/-/blob/amd-staging-drm-next/drive...
Ok.
I would totally agree that the driver does not want to be analysing LUT entries to decipher if it could use a fixed-function element or not. It would introduce uncertainty in the UAPI. So fixed-function elements would need their own properties, but I don't know if that is feasible as generic UAPI or if it should be driver-specific (and so left unused by generic userspace).
For the CRTC LUTs we actually do a linearity check to program the HW into bypass when the LUT is linear since the KMS LUT definition doesn't map well onto the LUT definition used by our HW and leads to rounding errors and failing IGT kms_color tests (if I remember this correctly).
https://gitlab.freedesktop.org/agd5f/linux/-/blob/amd-staging-drm-next/drive...
Hence the suggestion to define pre-defined TFs right at a KMS level for usecases where we can assume the display will tonemap the content.
Right. Explaining this would have been a good introduction in your cover letter.
Maybe you want to define new KMS properties that shall be mutually exclusive with the existing KMS GAMMA/CTM/DEGAMMA properties and clearly document them as such.
Thanks, pq
From: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com
Due to the way displays and human vision work it is most effective to encode luminance information in a non-linear space.
For SDR this non-linear mapping is assumed to roughly use a gamma 2.2 curve. This was due to the way CRTs worked and was fine for SDR content with a low luminance range.
The large luminance range (0-10,000 nits) for HDR exposes some short-comings of a simple gamma curve that have been addressed through various Electro-Optical Transfer Functions (EOTFs).
Rather than assuming how framebuffer content is encoded we want to make sure userspace presenting HDR content is explicit about the EOTF of the content, so a driver can decide whether the content can be supported or not.
This Patch adds common color transfer functions for SDR/HDR. These can be used to communicate with the driver regarding the transformation to use for a given plane.
enums added: DRM_COLOR_TF_SRGB roughly 2.4 gamma with initial linear section DRM_COLOR_TF_BT709 Similar to Gamma 2.2-2.8 DRM_COLOR_TF_PQ2084 most common tf used for HDR video (HDR10/Dolby). Can support up to 10,000 nits
The usage is similar to color_encoding and color_range where the driver can specify the default and supported tfs and pass it into drm_plane_create_color_properties().
Signed-off-by: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com Signed-off-by: Harry Wentland harry.wentland@amd.com --- .../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 4 +- .../gpu/drm/arm/display/komeda/komeda_plane.c | 4 +- drivers/gpu/drm/arm/malidp_planes.c | 4 +- drivers/gpu/drm/armada/armada_overlay.c | 4 +- drivers/gpu/drm/drm_atomic_uapi.c | 4 ++ drivers/gpu/drm/drm_color_mgmt.c | 63 +++++++++++++++++-- drivers/gpu/drm/i915/display/intel_sprite.c | 4 +- .../drm/i915/display/skl_universal_plane.c | 4 +- drivers/gpu/drm/nouveau/dispnv04/overlay.c | 4 +- drivers/gpu/drm/omapdrm/omap_plane.c | 4 +- drivers/gpu/drm/sun4i/sun8i_vi_layer.c | 4 +- drivers/gpu/drm/tidss/tidss_plane.c | 6 +- include/drm/drm_color_mgmt.h | 15 ++++- include/drm/drm_plane.h | 16 +++++ 14 files changed, 124 insertions(+), 16 deletions(-)
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c index 6985362c367d..28481540327f 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c @@ -7127,7 +7127,9 @@ static int amdgpu_dm_plane_init(struct amdgpu_display_manager *dm, BIT(DRM_COLOR_YCBCR_BT2020), BIT(DRM_COLOR_YCBCR_LIMITED_RANGE) | BIT(DRM_COLOR_YCBCR_FULL_RANGE), - DRM_COLOR_YCBCR_BT709, DRM_COLOR_YCBCR_LIMITED_RANGE); + BIT(DRM_COLOR_TF_SRGB), + DRM_COLOR_YCBCR_BT709, DRM_COLOR_YCBCR_LIMITED_RANGE, + DRM_COLOR_TF_SRGB); }
supported_rotations = diff --git a/drivers/gpu/drm/arm/display/komeda/komeda_plane.c b/drivers/gpu/drm/arm/display/komeda/komeda_plane.c index 2d5066ea270c..eb8ccf805408 100644 --- a/drivers/gpu/drm/arm/display/komeda/komeda_plane.c +++ b/drivers/gpu/drm/arm/display/komeda/komeda_plane.c @@ -299,8 +299,10 @@ static int komeda_plane_add(struct komeda_kms_dev *kms, BIT(DRM_COLOR_YCBCR_BT2020), BIT(DRM_COLOR_YCBCR_LIMITED_RANGE) | BIT(DRM_COLOR_YCBCR_FULL_RANGE), + BIT(DRM_COLOR_TF_SRGB), DRM_COLOR_YCBCR_BT601, - DRM_COLOR_YCBCR_LIMITED_RANGE); + DRM_COLOR_YCBCR_LIMITED_RANGE, + DRM_COLOR_TF_SRGB); if (err) goto cleanup;
diff --git a/drivers/gpu/drm/arm/malidp_planes.c b/drivers/gpu/drm/arm/malidp_planes.c index 351a85088d0e..0d77a2c0c829 100644 --- a/drivers/gpu/drm/arm/malidp_planes.c +++ b/drivers/gpu/drm/arm/malidp_planes.c @@ -1020,7 +1020,9 @@ int malidp_de_planes_init(struct drm_device *drm) BIT(DRM_COLOR_YCBCR_BT2020), BIT(DRM_COLOR_YCBCR_LIMITED_RANGE) | \ BIT(DRM_COLOR_YCBCR_FULL_RANGE), - enc, range); + BIT(DRM_COLOR_SRGB), + enc, range, + DRM_COLOR_SRGB); if (!ret) /* program the HW registers */ malidp_de_set_color_encoding(plane, enc, range); diff --git a/drivers/gpu/drm/armada/armada_overlay.c b/drivers/gpu/drm/armada/armada_overlay.c index 6346b890279a..ad3a743242f0 100644 --- a/drivers/gpu/drm/armada/armada_overlay.c +++ b/drivers/gpu/drm/armada/armada_overlay.c @@ -589,8 +589,10 @@ int armada_overlay_plane_create(struct drm_device *dev, unsigned long crtcs) BIT(DRM_COLOR_YCBCR_BT601) | BIT(DRM_COLOR_YCBCR_BT709), BIT(DRM_COLOR_YCBCR_LIMITED_RANGE), + BIT(DRM_COLOR_TF_SRGB), DEFAULT_ENCODING, - DRM_COLOR_YCBCR_LIMITED_RANGE); + DRM_COLOR_YCBCR_LIMITED_RANGE, + DRM_COLOR_TF_SRGB);
return ret; } diff --git a/drivers/gpu/drm/drm_atomic_uapi.c b/drivers/gpu/drm/drm_atomic_uapi.c index 268bb69c2e2f..ea95c1224253 100644 --- a/drivers/gpu/drm/drm_atomic_uapi.c +++ b/drivers/gpu/drm/drm_atomic_uapi.c @@ -595,6 +595,8 @@ static int drm_atomic_plane_set_property(struct drm_plane *plane, state->color_encoding = val; } else if (property == plane->color_range_property) { state->color_range = val; + } else if (property == plane->color_tf_property) { + state->color_tf = val; } else if (property == config->prop_fb_damage_clips) { ret = drm_atomic_replace_property_blob_from_id(dev, &state->fb_damage_clips, @@ -661,6 +663,8 @@ drm_atomic_plane_get_property(struct drm_plane *plane, *val = state->color_encoding; } else if (property == plane->color_range_property) { *val = state->color_range; + } else if (property == plane->color_tf_property) { + *val = state->color_tf; } else if (property == config->prop_fb_damage_clips) { *val = (state->fb_damage_clips) ? state->fb_damage_clips->base.id : 0; diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c index a183ebae2941..2404b07046c5 100644 --- a/drivers/gpu/drm/drm_color_mgmt.c +++ b/drivers/gpu/drm/drm_color_mgmt.c @@ -106,6 +106,11 @@ * Optional plane enum property to support different non RGB * color parameter ranges. The driver can provide a subset of * standard enum values supported by the DRM plane. + * + * "COLOR_TRANFER_FUNCTION": + * Optional plane enum property to support different + * color luminance mappings. The driver can provide a subset of + * standard enum values supported by the DRM plane. */
/** @@ -480,6 +485,10 @@ static const char * const color_range_name[] = { [DRM_COLOR_YCBCR_LIMITED_RANGE] = "YCbCr limited range", };
+static const char * const color_tf_name[] = { + [DRM_COLOR_TF_SRGB] = "sRGB", + [DRM_COLOR_TF_PQ2084] = "PQ2084", +}; /** * drm_get_color_encoding_name - return a string for color encoding * @encoding: color encoding to compute name of @@ -510,30 +519,49 @@ const char *drm_get_color_range_name(enum drm_color_range range) return color_range_name[range]; }
+/** + * drm_get_color_transfer_function - return a string for color transfer function + * @tf: transfer function to compute name of + * + * In contrast to the other drm_get_*_name functions this one here returns a + * const pointer and hence is threadsafe. + */ +const char *drm_get_color_transfer_function_name(enum drm_color_transfer_function tf) +{ + if (WARN_ON(tf >= ARRAY_SIZE(color_tf_name))) + return "unknown"; + + return color_tf_name[tf]; +} /** * drm_plane_create_color_properties - color encoding related plane properties * @plane: plane object * @supported_encodings: bitfield indicating supported color encodings * @supported_ranges: bitfileld indicating supported color ranges + * @supported_tfs: bitfileld indicating supported color transfer functions * @default_encoding: default color encoding * @default_range: default color range + * @default_tf: default color transfer function * - * Create and attach plane specific COLOR_ENCODING and COLOR_RANGE - * properties to @plane. The supported encodings and ranges should - * be provided in supported_encodings and supported_ranges bitmasks. + * Create and attach plane specific COLOR_ENCODING, COLOR_RANGE and COLOR_TRANSFER_FUNCTION + * properties to @plane. The supported encodings, ranges and tfs should + * be provided in supported_encodings, supported_ranges and supported_tfs bitmasks. * Each bit set in the bitmask indicates that its number as enum * value is supported. */ int drm_plane_create_color_properties(struct drm_plane *plane, u32 supported_encodings, u32 supported_ranges, + u32 supported_tfs, enum drm_color_encoding default_encoding, - enum drm_color_range default_range) + enum drm_color_range default_range, + enum drm_color_transfer_function default_tf) { struct drm_device *dev = plane->dev; struct drm_property *prop; struct drm_prop_enum_list enum_list[max_t(int, DRM_COLOR_ENCODING_MAX, - DRM_COLOR_RANGE_MAX)]; + max_t(int, DRM_COLOR_RANGE_MAX, + DRM_COLOR_TF_MAX))]; int i, len;
if (WARN_ON(supported_encodings == 0 || @@ -546,6 +574,11 @@ int drm_plane_create_color_properties(struct drm_plane *plane, (supported_ranges & BIT(default_range)) == 0)) return -EINVAL;
+ if (WARN_ON(supported_tfs == 0 || + (supported_tfs & -BIT(DRM_COLOR_TF_MAX)) != 0 || + (supported_tfs & BIT(default_tf)) == 0)) + return -EINVAL; + len = 0; for (i = 0; i < DRM_COLOR_ENCODING_MAX; i++) { if ((supported_encodings & BIT(i)) == 0) @@ -584,6 +617,26 @@ int drm_plane_create_color_properties(struct drm_plane *plane, if (plane->state) plane->state->color_range = default_range;
+ + len = 0; + for (i = 0; i < DRM_COLOR_TF_MAX; i++) { + if ((supported_tfs & BIT(i)) == 0) + continue; + + enum_list[len].type = i; + enum_list[len].name = color_tf_name[i]; + len++; + } + + prop = drm_property_create_enum(dev, 0, "COLOR_TRANSFER_FUNCTION", + enum_list, len); + if (!prop) + return -ENOMEM; + plane->color_tf_property = prop; + drm_object_attach_property(&plane->base, prop, default_tf); + if (plane->state) + plane->state->color_tf = default_tf; + return 0; } EXPORT_SYMBOL(drm_plane_create_color_properties); diff --git a/drivers/gpu/drm/i915/display/intel_sprite.c b/drivers/gpu/drm/i915/display/intel_sprite.c index 4cbdb8fd4bb1..2208c3d82246 100644 --- a/drivers/gpu/drm/i915/display/intel_sprite.c +++ b/drivers/gpu/drm/i915/display/intel_sprite.c @@ -1881,8 +1881,10 @@ intel_sprite_plane_create(struct drm_i915_private *dev_priv, BIT(DRM_COLOR_YCBCR_BT709), BIT(DRM_COLOR_YCBCR_LIMITED_RANGE) | BIT(DRM_COLOR_YCBCR_FULL_RANGE), + BIT(DRM_COLOR_TF_SRGB), DRM_COLOR_YCBCR_BT709, - DRM_COLOR_YCBCR_LIMITED_RANGE); + DRM_COLOR_YCBCR_LIMITED_RANGE, + DRM_COLOR_TF_SRGB);
zpos = sprite + 1; drm_plane_create_zpos_immutable_property(&plane->base, zpos); diff --git a/drivers/gpu/drm/i915/display/skl_universal_plane.c b/drivers/gpu/drm/i915/display/skl_universal_plane.c index 1f335cb09149..c2d64ef9ae3b 100644 --- a/drivers/gpu/drm/i915/display/skl_universal_plane.c +++ b/drivers/gpu/drm/i915/display/skl_universal_plane.c @@ -2091,8 +2091,10 @@ skl_universal_plane_create(struct drm_i915_private *dev_priv, supported_csc, BIT(DRM_COLOR_YCBCR_LIMITED_RANGE) | BIT(DRM_COLOR_YCBCR_FULL_RANGE), + BIT(DRM_COLOR_TF_SRGB), DRM_COLOR_YCBCR_BT709, - DRM_COLOR_YCBCR_LIMITED_RANGE); + DRM_COLOR_YCBCR_LIMITED_RANGE, + DRM_COLOR_TF_SRGB);
drm_plane_create_alpha_property(&plane->base); drm_plane_create_blend_mode_property(&plane->base, diff --git a/drivers/gpu/drm/nouveau/dispnv04/overlay.c b/drivers/gpu/drm/nouveau/dispnv04/overlay.c index 37e63e98cd08..fe40e24469d1 100644 --- a/drivers/gpu/drm/nouveau/dispnv04/overlay.c +++ b/drivers/gpu/drm/nouveau/dispnv04/overlay.c @@ -345,8 +345,10 @@ nv10_overlay_init(struct drm_device *device) BIT(DRM_COLOR_YCBCR_BT601) | BIT(DRM_COLOR_YCBCR_BT709), BIT(DRM_COLOR_YCBCR_LIMITED_RANGE), + BIT(DRM_COLOR_TF_SRGB), DRM_COLOR_YCBCR_BT601, - DRM_COLOR_YCBCR_LIMITED_RANGE); + DRM_COLOR_YCBCR_LIMITED_RANGE, + DRM_COLOR_TF_SRGB);
plane->set_params = nv10_set_params; nv10_set_params(plane); diff --git a/drivers/gpu/drm/omapdrm/omap_plane.c b/drivers/gpu/drm/omapdrm/omap_plane.c index 51dc24acea73..0929e1c39653 100644 --- a/drivers/gpu/drm/omapdrm/omap_plane.c +++ b/drivers/gpu/drm/omapdrm/omap_plane.c @@ -319,8 +319,10 @@ struct drm_plane *omap_plane_init(struct drm_device *dev, BIT(DRM_COLOR_YCBCR_BT709), BIT(DRM_COLOR_YCBCR_FULL_RANGE) | BIT(DRM_COLOR_YCBCR_LIMITED_RANGE), + BIT(DRM_COLOR_TF_SRGB), DRM_COLOR_YCBCR_BT601, - DRM_COLOR_YCBCR_FULL_RANGE); + DRM_COLOR_YCBCR_FULL_RANGE, + DRM_COLOR_TF_SRGB);
return plane;
diff --git a/drivers/gpu/drm/sun4i/sun8i_vi_layer.c b/drivers/gpu/drm/sun4i/sun8i_vi_layer.c index 8abb59e2f0c0..d15267db2274 100644 --- a/drivers/gpu/drm/sun4i/sun8i_vi_layer.c +++ b/drivers/gpu/drm/sun4i/sun8i_vi_layer.c @@ -584,8 +584,10 @@ struct sun8i_vi_layer *sun8i_vi_layer_init_one(struct drm_device *drm, ret = drm_plane_create_color_properties(&layer->plane, supported_encodings, supported_ranges, + BIT(DRM_COLOR_TF_SRGB), DRM_COLOR_YCBCR_BT709, - DRM_COLOR_YCBCR_LIMITED_RANGE); + DRM_COLOR_YCBCR_LIMITED_RANGE, + DRM_COLOR_TF_SRGB); if (ret) { dev_err(drm->dev, "Couldn't add encoding and range properties!\n"); return ERR_PTR(ret); diff --git a/drivers/gpu/drm/tidss/tidss_plane.c b/drivers/gpu/drm/tidss/tidss_plane.c index 35067ae674ea..e50857cfd212 100644 --- a/drivers/gpu/drm/tidss/tidss_plane.c +++ b/drivers/gpu/drm/tidss/tidss_plane.c @@ -179,8 +179,10 @@ struct tidss_plane *tidss_plane_create(struct tidss_device *tidss, BIT(DRM_COLOR_YCBCR_BT709)); u32 color_ranges = (BIT(DRM_COLOR_YCBCR_FULL_RANGE) | BIT(DRM_COLOR_YCBCR_LIMITED_RANGE)); + u32 color_tfs = BIT(DRM_COLOR_TF_SRGB; u32 default_encoding = DRM_COLOR_YCBCR_BT601; u32 default_range = DRM_COLOR_YCBCR_FULL_RANGE; + u32 default_tf = DRM_COLOR_TF_SRGB;; u32 blend_modes = (BIT(DRM_MODE_BLEND_PREMULTI) | BIT(DRM_MODE_BLEND_COVERAGE)); int ret; @@ -210,8 +212,10 @@ struct tidss_plane *tidss_plane_create(struct tidss_device *tidss, ret = drm_plane_create_color_properties(&tplane->plane, color_encodings, color_ranges, + color_tfs, default_encoding, - default_range); + default_range, + default_tf); if (ret) goto err;
diff --git a/include/drm/drm_color_mgmt.h b/include/drm/drm_color_mgmt.h index 3043dd73480c..f59806366855 100644 --- a/include/drm/drm_color_mgmt.h +++ b/include/drm/drm_color_mgmt.h @@ -91,11 +91,24 @@ enum drm_color_range { DRM_COLOR_RANGE_MAX, };
+/** + * enum drm_color_transfer_function - common transfer function used for sdr/hdr formats + * + * DRM_COLOR_TF_SRGB - Based on gamma curve and is used for printer/monitors/web + * DRM_COLOR_TF_PQ2084 - Used for HDR and allows for up to 10,000 nit support. +*/ +enum drm_color_transfer_function { + DRM_COLOR_TF_SRGB, + DRM_COLOR_TF_PQ2084, + DRM_COLOR_TF_MAX, +}; int drm_plane_create_color_properties(struct drm_plane *plane, u32 supported_encodings, u32 supported_ranges, + u32 supported_tf, enum drm_color_encoding default_encoding, - enum drm_color_range default_range); + enum drm_color_range default_range, + enum drm_color_transfer_function default_tf);
/** * enum drm_color_lut_tests - hw-specific LUT tests to perform diff --git a/include/drm/drm_plane.h b/include/drm/drm_plane.h index 95ab14a4336a..c85c59501a7a 100644 --- a/include/drm/drm_plane.h +++ b/include/drm/drm_plane.h @@ -179,6 +179,14 @@ struct drm_plane_state { */ enum drm_color_range color_range;
+ /** + * @color_transfer_function: + * + * Transfer function for HDR color/luminance mapping. This will allow the + * driver to know what transfer function should be used to for the current + * format for a proper HDR color/luminance output. + */ + enum drm_color_transfer_function color_tf; /** * @fb_damage_clips: * @@ -741,6 +749,14 @@ struct drm_plane { * See drm_plane_create_color_properties(). */ struct drm_property *color_range_property; + /** + * @color_tf_property: + * + * Optional "COLOR_TRANSFER_FUNCTION" enum property for specifying + * color transfer function for non RGB formats, mostly used for HDR. + * See drm_plane_create_color_properties(). + */ + struct drm_property *color_tf_property;
/** * @scaling_filter_property: property to apply a particular filter while
From: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com
SDR is typically mastered at 200 nits and HDR is mastered at up to 10,000 nits. Due to this luminance range difference if we blend a SDR and HDR plane together, we can run into problems where the HDR plane is too bright or the SDR plane is too dim
A common solution to this problem is to boost the SDR plane so that its not too dim.
This patch introduces a "sdr_white_level" property, this property can be used by the userspace to boost the SDR content luminance. The boost value is a explicit luiminance value in nits. This allows the userspace to set the maximum white level for the SDR plane.
Signed-off-by: Bhawanpreet Lakha Bhawanpreet.Lakha@amd.com Signed-off-by: Harry Wentland harry.wentland@amd.com --- drivers/gpu/drm/drm_atomic_uapi.c | 4 ++++ drivers/gpu/drm/drm_color_mgmt.c | 17 +++++++++++++++++ include/drm/drm_color_mgmt.h | 6 ++++++ include/drm/drm_plane.h | 14 ++++++++++++++ 4 files changed, 41 insertions(+)
diff --git a/drivers/gpu/drm/drm_atomic_uapi.c b/drivers/gpu/drm/drm_atomic_uapi.c index ea95c1224253..b3b6de7b74aa 100644 --- a/drivers/gpu/drm/drm_atomic_uapi.c +++ b/drivers/gpu/drm/drm_atomic_uapi.c @@ -597,6 +597,8 @@ static int drm_atomic_plane_set_property(struct drm_plane *plane, state->color_range = val; } else if (property == plane->color_tf_property) { state->color_tf = val; + } else if (property == plane->sdr_white_level_property) { + state->sdr_white_level = val; } else if (property == config->prop_fb_damage_clips) { ret = drm_atomic_replace_property_blob_from_id(dev, &state->fb_damage_clips, @@ -665,6 +667,8 @@ drm_atomic_plane_get_property(struct drm_plane *plane, *val = state->color_range; } else if (property == plane->color_tf_property) { *val = state->color_tf; + } else if (property == plane->sdr_white_level_property) { + *val = state->sdr_white_level; } else if (property == config->prop_fb_damage_clips) { *val = (state->fb_damage_clips) ? state->fb_damage_clips->base.id : 0; diff --git a/drivers/gpu/drm/drm_color_mgmt.c b/drivers/gpu/drm/drm_color_mgmt.c index 2404b07046c5..a0b77d7d0565 100644 --- a/drivers/gpu/drm/drm_color_mgmt.c +++ b/drivers/gpu/drm/drm_color_mgmt.c @@ -519,6 +519,23 @@ const char *drm_get_color_range_name(enum drm_color_range range) return color_range_name[range]; }
+int drm_plane_create_sdr_white_level_property(struct drm_plane *plane){ + + struct drm_property *prop; + + prop = drm_property_create_range(plane->dev, 0, "SDR_WHITE_LEVEL", 0, UINT_MAX); + + if (!prop) + return -ENOMEM; + + plane->sdr_white_level_property = prop; + drm_object_attach_property(&plane->base, prop, DRM_DEFAULT_SDR_WHITE_LEVEL); + + if (plane->state) + plane->state->sdr_white_level = DRM_DEFAULT_SDR_WHITE_LEVEL; + + return 0; +} /** * drm_get_color_transfer_function - return a string for color transfer function * @tf: transfer function to compute name of diff --git a/include/drm/drm_color_mgmt.h b/include/drm/drm_color_mgmt.h index f59806366855..a020346b1747 100644 --- a/include/drm/drm_color_mgmt.h +++ b/include/drm/drm_color_mgmt.h @@ -26,6 +26,12 @@ #include <linux/ctype.h> #include <drm/drm_property.h>
+/** + * Default SDR white level in nits. Although there is no standard SDR nit level, 200 + * is chosen as the default since that is the generally accepted value. + */ +#define DRM_DEFAULT_SDR_WHITE_LEVEL 200 + struct drm_crtc; struct drm_plane;
diff --git a/include/drm/drm_plane.h b/include/drm/drm_plane.h index c85c59501a7a..fad8b7dd430c 100644 --- a/include/drm/drm_plane.h +++ b/include/drm/drm_plane.h @@ -187,6 +187,11 @@ struct drm_plane_state { * format for a proper HDR color/luminance output. */ enum drm_color_transfer_function color_tf; + /** + * @sdr_white_level: + * SDR white level boost for HDR+SDR multi plane usecases. max white level in nits + */ + unsigned int sdr_white_level; /** * @fb_damage_clips: * @@ -757,6 +762,15 @@ struct drm_plane { * See drm_plane_create_color_properties(). */ struct drm_property *color_tf_property; + /** + * @sdr_white_level: + * + * Optional sdr_white_level. When HDR and SDR are combined in multi plane + * overlay cases, the sdr plane will be very dim. This property allows + * the driver to boost the sdr plane's white level. The value should be + * max white level in nits. + */ + struct drm_property *sdr_white_level_property;
/** * @scaling_filter_property: property to apply a particular filter while
On Mon, Apr 26, 2021 at 01:38:49PM -0400, Harry Wentland wrote:
## Introduction
We are looking to enable HDR support for a couple of single-plane and multi-plane scenarios. To do this effectively we recommend new interfaces to drm_plane. Below I'll give a bit of background on HDR and why we propose these interfaces.
I think this is on of the topics that would tremendously benefit from the uapi rfc process, with lots of compositor people involved.
https://dri.freedesktop.org/docs/drm/gpu/rfc/
Also for this I think we really do need a pretty solid understanding of the involve compositor protocols, otherwise the kernel uapi is going to be for naught. -Daniel
## Defining a pixel's luminance
Currently the luminance space of pixels in a framebuffer/plane presented to the display is not well defined. It's usually assumed to be in a 2.2 or 2.4 gamma space and has no mapping to an absolute luminance value but is interpreted in relative terms.
Luminance can be measured and described in absolute terms as candela per meter squared, or cd/m2, or nits. Even though a pixel value can be mapped to luminance in a linear fashion to do so without losing a lot of detail requires 16-bpc color depth. The reason for this is that human perception can distinguish roughly between a 0.5-1% luminance delta. A linear representation is suboptimal, wasting precision in the highlights and losing precision in the shadows.
A gamma curve is a decent approximation to a human's perception of luminance, but the PQ (perceptual quantizer) function [1] improves on it. It also defines the luminance values in absolute terms, with the highest value being 10,000 nits and the lowest 0.0005 nits.
Using a content that's defined in PQ space we can approximate the real world in a much better way.
Here are some examples of real-life objects and their approximate luminance values:
| Object | Luminance in nits | | ----------------- | ----------------- | | Sun | 1.6 million | | Fluorescent light | 10,000 | | Highlights | 1,000 - sunlight | | White Objects | 250 - 1,000 | | Typical objects | 1 - 250 | | Shadows | 0.01 - 1 | | Ultra Blacks | 0 - 0.0005 |
## Describing the luminance space
**We propose a new drm_plane property to describe the Eletro-Optical Transfer Function (EOTF) with which its framebuffer was composed.** Examples of EOTF are:
| EOTF | Description | | --------- |:------------------------------------------------------------------------- | | Gamma 2.2 | a simple 2.2 gamma | | sRGB | 2.4 gamma with small initial linear section | | PQ 2084 | SMPTE ST 2084; used for HDR video and allows for up to 10,000 nit support | | Linear | Linear relationship between pixel value and luminance value |
## Mastering Luminances
Now we are able to use the PQ 2084 EOTF to define the luminance of pixels in absolute terms. Unfortunately we're again presented with physical limitations of the display technologies on the market today. Here are a few examples of luminance ranges of displays.
| Display | Luminance range in nits | | ------------------------ | ----------------------- | | Typical PC display | 0.3 - 200 | | Excellent LCD HDTV | 0.3 - 400 | | HDR LCD w/ local dimming | 0.05 - 1,500 |
Since no display can currently show the full 0.0005 to 10,000 nits luminance range the display will need to tonemap the HDR content, i.e to fit the content within a display's capabilities. To assist with tonemapping HDR content is usually accompanied with a metadata that describes (among other things) the minimum and maximum mastering luminance, i.e. the maximum and minimum luminance of the display that was used to master the HDR content.
The HDR metadata is currently defined on the drm_connector via the hdr_output_metadata blob property.
It might be useful to define per-plane hdr metadata, as different planes might have been mastered differently.
## SDR Luminance
Since SDR covers a smaller luminance range than HDR, an SDR plane might look dark when blended with HDR content. Since the max HDR luminance can be quite variable (200-1,500 nits on actual displays) it is best to make the SDR maximum luminance value configurable.
**We propose a drm_plane property to specfy the desired maximum luminance of the SDR plane in nits.** This allows us to map the SDR content predictably into HDR's absolute luminance space.
## Let There Be Color
So far we've only talked about luminance, ignoring colors altogether. Just like in the luminance space, traditionally the color space of display outputs has not been well defined. Similar to how an EOTF defines a mapping of pixel data to an absolute luminance value, the color space maps color information for each pixel onto the CIE 1931 chromaticity space. This can be thought of as a mapping to an absolute, real-life, color value.
A color space is defined by its primaries and white point. The primaries and white point are expressed as coordinates in the CIE 1931 color space. Think of the red primary as the reddest red that can be displayed within the color space. Same for green and blue.
Examples of color spaces are:
| Color Space | Description | | ----------- | ------------------------------------------ | | BT 601 | similar to BT 709 | | BT 709 | used by sRGB content; ~53% of BT 2020 | | DCI-P3 | used by most HDR displays; ~72% of BT 2020 | | BT 2020 | standard for most HDR content |
The color space is defined in DRM for YCbCr planes via the color_encoding property of the drm_plane.
**We propose to add definitions for the RGB variants of the BT color spaces.**
## Color Primaries and White Point
Just like displays can currently not represent the entire 0.0005 - 10,000 nits HDR range of the PQ 2084 EOTF, they are currently not capable of representing the entire BT.2020 color Gamut. For this reason video content will often specify the color primaries and white point used to master the video, in order to allow displays to be able to map the image as best as possible onto the display's gamut.
## Displays and Tonemapping
External displays are able to do their own tone and color mapping, based on the mastering luminance, color primaries, and white space defined in the HDR metadata.
Internal panels (which are currently few and far between) usually don't include the complex HW to do tone and color mapping on their own and will require the display driver to perform appropriate mapping.
## Pixel Formats
The pixel formats, such as ARGB8888, ARGB2101010, P010, or FP16 are unrelated to color space and EOTF definitions. HDR pixels can be formatted in different ways but in order to not lose precision HDR content requires at least 10 bpc precision. For this reason ARGB2101010, P010, and FP16 are the obvious candidates for HDR. ARGB2101010 and P010 have the advantage of requiring only half the bandwidth as FP16, while FP16 has the advantage of enough precision to operate in a linear space, i.e. without EOTF.
## Proposed use-cases
Although the userspace side of this work is still in the early stages it is clear that we will want to support the following two use-cases:
**One XRGB2101010 HDR Plane:** A single, composited plane of HDR content. The use-case is a video player on a desktop with the compositor owning the composition of SDR and HDR content. The content shall be PQ BT.2020 formatted. The drm_connector's hdr_output_metadata shall be set.
**One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc desktop plane, with a P010 HDR video plane underlayed. The HDR plane shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR boost value. The drm_connector's hdr_output_metadata shall be set.
**One XRGB8888 SDR Plane - HDR output:** In order to support a smooth transition we recommend an OS that supports HDR output to provide the hdr_output_metadata on the drm_connector to configure the output for HDR, even when the content is only SDR. This will allow for a smooth transition between SDR-only and HDR content. In this use-case the SDR max luminance value should be provided on the drm_plane.
In DCN we will de-PQ or de-Gamma all input in order to blend in linear space. For SDR content we will also apply any desired boost before blending. After blending we will then re-apply the PQ EOTF and do RGB to YCbCr conversion if needed.
## Summary of proposed interface changes
per drm_plane:
- new RGB color space definitions, mirroring the existing YUV color space definitions
- new transfer function property
- new SDR maximum white level property
## References
[1] https://en.wikipedia.org/wiki/High-dynamic-range_video#Perceptual_Quantizer
## Further Reading
https://gitlab.freedesktop.org/swick/wayland-protocols/-/blob/color/unstable... http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP309.pdf https://app.spectracal.com/Documents/White%20Papers/HDR_Demystified.pdf
Bhawanpreet Lakha (3): drm/color: Add RGB Color encodings drm/color: Add Color transfer functions for HDR/SDR drm/color: Add sdr boost property
.../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 4 +- .../gpu/drm/arm/display/komeda/komeda_plane.c | 4 +- drivers/gpu/drm/arm/malidp_planes.c | 4 +- drivers/gpu/drm/armada/armada_overlay.c | 4 +- drivers/gpu/drm/drm_atomic_uapi.c | 8 ++ drivers/gpu/drm/drm_color_mgmt.c | 84 +++++++++++++++++-- drivers/gpu/drm/i915/display/intel_sprite.c | 4 +- .../drm/i915/display/skl_universal_plane.c | 4 +- drivers/gpu/drm/nouveau/dispnv04/overlay.c | 4 +- drivers/gpu/drm/omapdrm/omap_plane.c | 4 +- drivers/gpu/drm/sun4i/sun8i_vi_layer.c | 4 +- drivers/gpu/drm/tidss/tidss_plane.c | 6 +- include/drm/drm_color_mgmt.h | 25 +++++- include/drm/drm_plane.h | 30 +++++++ 14 files changed, 173 insertions(+), 16 deletions(-)
-- 2.31.0
dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel
On Mon, 26 Apr 2021 13:38:49 -0400 Harry Wentland harry.wentland@amd.com wrote:
## Introduction
We are looking to enable HDR support for a couple of single-plane and multi-plane scenarios. To do this effectively we recommend new interfaces to drm_plane. Below I'll give a bit of background on HDR and why we propose these interfaces.
## Defining a pixel's luminance
Currently the luminance space of pixels in a framebuffer/plane presented to the display is not well defined. It's usually assumed to be in a 2.2 or 2.4 gamma space and has no mapping to an absolute luminance value but is interpreted in relative terms.
Luminance can be measured and described in absolute terms as candela per meter squared, or cd/m2, or nits. Even though a pixel value can be mapped to luminance in a linear fashion to do so without losing a lot of detail requires 16-bpc color depth. The reason for this is that human perception can distinguish roughly between a 0.5-1% luminance delta. A linear representation is suboptimal, wasting precision in the highlights and losing precision in the shadows.
A gamma curve is a decent approximation to a human's perception of luminance, but the PQ (perceptual quantizer) function [1] improves on it. It also defines the luminance values in absolute terms, with the highest value being 10,000 nits and the lowest 0.0005 nits.
Using a content that's defined in PQ space we can approximate the real world in a much better way.
Here are some examples of real-life objects and their approximate luminance values:
| Object | Luminance in nits | | ----------------- | ----------------- | | Sun | 1.6 million | | Fluorescent light | 10,000 | | Highlights | 1,000 - sunlight | | White Objects | 250 - 1,000 | | Typical objects | 1 - 250 | | Shadows | 0.01 - 1 | | Ultra Blacks | 0 - 0.0005 |
## Describing the luminance space
**We propose a new drm_plane property to describe the Eletro-Optical Transfer Function (EOTF) with which its framebuffer was composed.** Examples of EOTF are:
| EOTF | Description | | --------- |:------------------------------------------------------------------------- | | Gamma 2.2 | a simple 2.2 gamma | | sRGB | 2.4 gamma with small initial linear section | | PQ 2084 | SMPTE ST 2084; used for HDR video and allows for up to 10,000 nit support | | Linear | Linear relationship between pixel value and luminance value |
The definitions agree with what I have learnt so far. However, with these EOTF definitions, only PQ defines absolute luminance values while the others do not. So this is not enough information to blend planes together if they do not all use the same EOTF with the same dynamic range. More below.
## Mastering Luminances
Now we are able to use the PQ 2084 EOTF to define the luminance of pixels in absolute terms. Unfortunately we're again presented with physical limitations of the display technologies on the market today. Here are a few examples of luminance ranges of displays.
| Display | Luminance range in nits | | ------------------------ | ----------------------- | | Typical PC display | 0.3 - 200 | | Excellent LCD HDTV | 0.3 - 400 | | HDR LCD w/ local dimming | 0.05 - 1,500 |
Since no display can currently show the full 0.0005 to 10,000 nits luminance range the display will need to tonemap the HDR content, i.e to fit the content within a display's capabilities. To assist with tonemapping HDR content is usually accompanied with a metadata that describes (among other things) the minimum and maximum mastering luminance, i.e. the maximum and minimum luminance of the display that was used to master the HDR content.
The HDR metadata is currently defined on the drm_connector via the hdr_output_metadata blob property.
It might be useful to define per-plane hdr metadata, as different planes might have been mastered differently.
I don't think this would directly help with the dynamic range blending problem. You still need to establish the mapping between the optical values from two different EOTFs and dynamic ranges. Or can you know which optical values match the mastering display maximum and minimum luminances for not-PQ?
## SDR Luminance
Since SDR covers a smaller luminance range than HDR, an SDR plane might look dark when blended with HDR content. Since the max HDR luminance can be quite variable (200-1,500 nits on actual displays) it is best to make the SDR maximum luminance value configurable.
**We propose a drm_plane property to specfy the desired maximum luminance of the SDR plane in nits.** This allows us to map the SDR content predictably into HDR's absolute luminance space.
What would be the mapping? Simple linear scaling? A more complicated tone mapping?
Rather than "SDR luminance", do you perhaps intend this to configure the dynamic range of the non-absolute-luminance EOTFs? In that case maybe you'd need a black luminance level too?
## Let There Be Color
So far we've only talked about luminance, ignoring colors altogether. Just like in the luminance space, traditionally the color space of display outputs has not been well defined. Similar to how an EOTF defines a mapping of pixel data to an absolute luminance value, the color space maps color information for each pixel onto the CIE 1931 chromaticity space. This can be thought of as a mapping to an absolute, real-life, color value.
A color space is defined by its primaries and white point. The primaries and white point are expressed as coordinates in the CIE 1931 color space. Think of the red primary as the reddest red that can be displayed within the color space. Same for green and blue.
Examples of color spaces are:
| Color Space | Description | | ----------- | ------------------------------------------ | | BT 601 | similar to BT 709 | | BT 709 | used by sRGB content; ~53% of BT 2020 | | DCI-P3 | used by most HDR displays; ~72% of BT 2020 | | BT 2020 | standard for most HDR content |
The color space is defined in DRM for YCbCr planes via the color_encoding property of the drm_plane.
I don't think that is quite right.
As far I understand, COLOR_ENCODING property controls which matrix is used to convert from YCbCr to RGB, but that is all it does. It is not used for the actual color space. So while these BT standards do specify the chromaticities, they also specify the YCbCr encoding which is the part used in this property.
YCbCr and RGB are color models. They are not color spaces. RGB is an additive color model while YCbCr is not, AFAIU. Blending requires an additive color model and linear luminance encoding.
You need two color space definitions to create one color space transformation: source color space and destination color space. You also need an idea *how* the two color spaces should be mapped, which is called "rendering intent". You can't do anything with just one color space definition, except to pass it on along with the pixels.
To be able to blend planes together, all planes need to be converted to the same color space first: the blending color space, whatever you choose it to be. I do not see where KMS would do this color space conversion, or where it would get the definition of the blending color space.
**We propose to add definitions for the RGB variants of the BT color spaces.**
Therefore I'm not sure this makes sense.
## Color Primaries and White Point
Just like displays can currently not represent the entire 0.0005 - 10,000 nits HDR range of the PQ 2084 EOTF, they are currently not capable of representing the entire BT.2020 color Gamut. For this reason video content will often specify the color primaries and white point used to master the video, in order to allow displays to be able to map the image as best as possible onto the display's gamut.
## Displays and Tonemapping
External displays are able to do their own tone and color mapping, based on the mastering luminance, color primaries, and white space defined in the HDR metadata.
Internal panels (which are currently few and far between) usually don't include the complex HW to do tone and color mapping on their own and will require the display driver to perform appropriate mapping.
FWIW, when designing Weston's color management, we are aiming for the latter "simple" panels foremost, because that gives us full control of all color conversions and tone mappings.
OTOH, if Weston has to present to a display which only accepts e.g. BT.2020/PQ signal, the display might always mangle the image in unexpected ways. Therefore I expect that by default Weston will do everything it can to try to make the display not apply anything magic image enhancement: trust that EDID description of the display gamut and dynamic range are correct, and use HDR metadata to tell the display that those values are exactly what we are using. And we use them.
IMO, a display doing its tone mapping magically is only useful when you want to be able to use "simple" playback devices that cannot adapt to the display they are driving. Magic tone mapping is also a way for hardware vendors to differentiate, which from the color management perspective is harmful as it makes it more difficult or impossible to predict the display behaviour or to keep it consistent.
So there are two opposing goals:
- Traditional color management wants absolute control of the display, leaving nothing unpredictable and preferably also nothing undefined. Undefined behaviour can always be measured (profiled) which makes it predictable and known. The viewing environment is controlled and constant.
- Entertainment wants the most visually impressive image quality by dynamically adapting to both displayed content and to the viewing environment conditions.
## Pixel Formats
The pixel formats, such as ARGB8888, ARGB2101010, P010, or FP16 are unrelated to color space and EOTF definitions. HDR pixels can be formatted in different ways but in order to not lose precision HDR content requires at least 10 bpc precision. For this reason ARGB2101010, P010, and FP16 are the obvious candidates for HDR. ARGB2101010 and P010 have the advantage of requiring only half the bandwidth as FP16, while FP16 has the advantage of enough precision to operate in a linear space, i.e. without EOTF.
Right.
## Proposed use-cases
Although the userspace side of this work is still in the early stages it is clear that we will want to support the following two use-cases:
**One XRGB2101010 HDR Plane:** A single, composited plane of HDR content. The use-case is a video player on a desktop with the compositor owning the composition of SDR and HDR content. The content shall be PQ BT.2020 formatted. The drm_connector's hdr_output_metadata shall be set.
This use case is already possible, right?
**One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc desktop plane, with a P010 HDR video plane underlayed. The HDR plane shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR boost value. The drm_connector's hdr_output_metadata shall be set.
This use case requires blending in KMS, so is the primary goal I suppose.
**One XRGB8888 SDR Plane - HDR output:** In order to support a smooth transition we recommend an OS that supports HDR output to provide the hdr_output_metadata on the drm_connector to configure the output for HDR, even when the content is only SDR. This will allow for a smooth transition between SDR-only and HDR content. In this use-case the SDR max luminance value should be provided on the drm_plane.
I think this might be already possible by crafting a CRTC GAMMA LUT? Not sure about precision.
In DCN we will de-PQ or de-Gamma all input in order to blend in linear space. For SDR content we will also apply any desired boost before blending. After blending we will then re-apply the PQ EOTF and do RGB to YCbCr conversion if needed.
This assumes the same color space over everything.
## Summary of proposed interface changes
per drm_plane:
- new RGB color space definitions, mirroring the existing YUV color
space definitions
- new transfer function property
- new SDR maximum white level property
How will these new KMS properties interact with per-plane DEGAMMA, CTM and/or GAMMA properties?
Why go with your proposal instead of per-plane CTM and LUT?
I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs both per-plane and on CRTC, would be:
plane: FB -> M1 -> LUT1 -> M2 -> blending input
CRTC: blending output -> LUT2 -> M3 -> connector
FB: framebuffer M1: matrix transform, capable of converting e.g. YCbCr to RGB LUT1: 1D LUT for content EOTF, to produce light-linear RGB M2: matrix transform for color space transformation
LUT2: 1D LUT for applying monitor EOTF^-1 M3: matrix transform, e.g. if you need to push YCbCr on the connector
We also need to know where and how clipping happens.
I think this scheme would allow implementing everything you want, and it would not be tied to rigid enumerations, and it won't have any magical conversions done under the hood as you would need to do to convert from one enum space to another. It leaves the render intent to be defined by the userspace compositor, rather than building a fixed policy in the kernel.
Userspace would be setting transformation operators, not color spaces, to the kernel, allowing the blending space to be chosen by userspace. In Weston we aim to choose then blending color space to be the same as the output color space, except in optical (linear) encoding. The output color space can be non-standard, e.g. measured with a display profiler equipment.
I would expect gamut mapping, dynamic range mapping and tone mapping to be places where most experimentation and innovation happens, so implementing them in the kernel with just few or no controllable parameters at this time seems like it could become useless fast.
Thanks, pq
## References
[1] https://en.wikipedia.org/wiki/High-dynamic-range_video#Perceptual_Quantizer
## Further Reading
https://gitlab.freedesktop.org/swick/wayland-protocols/-/blob/color/unstable... http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP309.pdf https://app.spectracal.com/Documents/White%20Papers/HDR_Demystified.pdf
Bhawanpreet Lakha (3): drm/color: Add RGB Color encodings drm/color: Add Color transfer functions for HDR/SDR drm/color: Add sdr boost property
.../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 4 +- .../gpu/drm/arm/display/komeda/komeda_plane.c | 4 +- drivers/gpu/drm/arm/malidp_planes.c | 4 +- drivers/gpu/drm/armada/armada_overlay.c | 4 +- drivers/gpu/drm/drm_atomic_uapi.c | 8 ++ drivers/gpu/drm/drm_color_mgmt.c | 84 +++++++++++++++++-- drivers/gpu/drm/i915/display/intel_sprite.c | 4 +- .../drm/i915/display/skl_universal_plane.c | 4 +- drivers/gpu/drm/nouveau/dispnv04/overlay.c | 4 +- drivers/gpu/drm/omapdrm/omap_plane.c | 4 +- drivers/gpu/drm/sun4i/sun8i_vi_layer.c | 4 +- drivers/gpu/drm/tidss/tidss_plane.c | 6 +- include/drm/drm_color_mgmt.h | 25 +++++- include/drm/drm_plane.h | 30 +++++++ 14 files changed, 173 insertions(+), 16 deletions(-)
Hello Harry,
Many of us in the mail chain have discussed this before, on what is the right way to blend and tone map a SDR and a HDR buffer from same/different color spaces, and what kind of DRM plane properties will be needed.
As you can see from the previous comments, that the majority of the decision making will happen in the Compositor, as it's the only SW unit, which has the overall picture clear.
Reference: (https://lists.freedesktop.org/archives/wayland-devel/2019-January/039808.htm... )
If we see a systematic approach of how do we make such blending policy, it will look like:
- Compositor needs to understand the following values of each of the buffer:
- Color space or Gamut: BT2020/SRGB/DCI-P3/BT709/BT601 etc
- Color format (RGB/YCBCR) and subsampling (444/422/420)
- Tone (SDR/HDR_A/HDR_B)
- Then the Compositor needs to understand the capabilities of the output display, as this will be a clamping value
- Output Gamut support (BT2020/SRGB/DCIP3)
- Output max Luminance of the monitor in Nits (even in case of HDR content to HDR display)
Based of all this information above, the compositor needs to set a blending target, which contains the following:
- Output Colorspace of the blended output: say BT2020
- Output Luminance of the blended output: Match content, if monitor can support it
- Output Color format of the blended output: Say YCBCR4:2:0
Let's assume compositor prepares a blending policy with output as:
- Output Luminance: HDR 500 Nits
- Output color space: BT2020
- Output color format: RGB888
- Output curve: ST2084
Assuming these details, A compositor will look for DRM color properties like these:
1. Degamma plane property : To make buffers linear for Gamut mapping
2. Gamut mapping plane property: To gamut map SRGB buffer to BT2020 colorspace
3. Color space conversion plane property: To convert from YCBCR->RGB
4. Tone mapping plane property: To tone map SDR buffer S2H and HDR buffer H2H
5. Gamma plane/CRTC property: to re-apply the output ST2084 curve
We will also need connector/CRTC properties to set AVI info-frames accordingly.
A high level block diagram for blending on a generic HW should look like this:
/* * SDR 200Nits┌────────────────┐ SDR 200 Nits ┌────────────────┐ SDR 200 ┌──────────────────┐HDR 500┌────────────────┐ HDR 500 * BT709 │ │ BT709 │ │ BT2020 │ │BT2020 │ │ BT2020 * ────────► │ Degamma ├─────────────► │ Gamut Mapping ├────────►│ Tone mapping ├──────►│ Gamma │ * RGB888 │ 2.2 │ RGB888 │ 709->2020 │ RGB888 │ S2H │RGB888 │ ST2084 │ RGB888 * Non Linear │ │ Linear │ │ Linear │ 200->500 │Linear │ │ ST2084 * └────────────────┘ └────────────────┘ └──────────────────┘ └────────────────┘ * * * * * * * * * ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌────────────────┐ * HDR 600 Nits│ │HDR 600 Nits │ │HDR600 │ │HDR500 │ │ HDR500 * ────────► │ Degamma ├────────────►│ Color space ├──────────►│ Tone mapping ├──────►│ Gamma │ * BT2020 │ OETF ST2084 │ BT2020 │ conversion │BT2020 │ H2H │BT2020 │ ST2084 │ BT2020 * YCBCR420 │ │ YCBCR420 │ YCBCR->RGB │RGB88 │ 600->500 │RGB888 │ │ RGB888 * Non Linear └─────────────────┘ Linear └─────────────────┘Linear └─────────────────┘Linear └────────────────┘ ST2084 */
Hope this helps to refine the series.
Regards
Shashank
On 27/04/21 20:20, Pekka Paalanen wrote:
On Mon, 26 Apr 2021 13:38:49 -0400 Harry Wentland harry.wentland@amd.com wrote:
## Introduction
We are looking to enable HDR support for a couple of single-plane and multi-plane scenarios. To do this effectively we recommend new interfaces to drm_plane. Below I'll give a bit of background on HDR and why we propose these interfaces.
## Defining a pixel's luminance
Currently the luminance space of pixels in a framebuffer/plane presented to the display is not well defined. It's usually assumed to be in a 2.2 or 2.4 gamma space and has no mapping to an absolute luminance value but is interpreted in relative terms.
Luminance can be measured and described in absolute terms as candela per meter squared, or cd/m2, or nits. Even though a pixel value can be mapped to luminance in a linear fashion to do so without losing a lot of detail requires 16-bpc color depth. The reason for this is that human perception can distinguish roughly between a 0.5-1% luminance delta. A linear representation is suboptimal, wasting precision in the highlights and losing precision in the shadows.
A gamma curve is a decent approximation to a human's perception of luminance, but the PQ (perceptual quantizer) function [1] improves on it. It also defines the luminance values in absolute terms, with the highest value being 10,000 nits and the lowest 0.0005 nits.
Using a content that's defined in PQ space we can approximate the real world in a much better way.
Here are some examples of real-life objects and their approximate luminance values:
| Object | Luminance in nits | | ----------------- | ----------------- | | Sun | 1.6 million | | Fluorescent light | 10,000 | | Highlights | 1,000 - sunlight | | White Objects | 250 - 1,000 | | Typical objects | 1 - 250 | | Shadows | 0.01 - 1 | | Ultra Blacks | 0 - 0.0005 |
## Describing the luminance space
**We propose a new drm_plane property to describe the Eletro-Optical Transfer Function (EOTF) with which its framebuffer was composed.** Examples of EOTF are:
| EOTF | Description | | --------- |:------------------------------------------------------------------------- | | Gamma 2.2 | a simple 2.2 gamma | | sRGB | 2.4 gamma with small initial linear section | | PQ 2084 | SMPTE ST 2084; used for HDR video and allows for up to 10,000 nit support | | Linear | Linear relationship between pixel value and luminance value |
The definitions agree with what I have learnt so far. However, with these EOTF definitions, only PQ defines absolute luminance values while the others do not. So this is not enough information to blend planes together if they do not all use the same EOTF with the same dynamic range. More below.
## Mastering Luminances
Now we are able to use the PQ 2084 EOTF to define the luminance of pixels in absolute terms. Unfortunately we're again presented with physical limitations of the display technologies on the market today. Here are a few examples of luminance ranges of displays.
| Display | Luminance range in nits | | ------------------------ | ----------------------- | | Typical PC display | 0.3 - 200 | | Excellent LCD HDTV | 0.3 - 400 | | HDR LCD w/ local dimming | 0.05 - 1,500 |
Since no display can currently show the full 0.0005 to 10,000 nits luminance range the display will need to tonemap the HDR content, i.e to fit the content within a display's capabilities. To assist with tonemapping HDR content is usually accompanied with a metadata that describes (among other things) the minimum and maximum mastering luminance, i.e. the maximum and minimum luminance of the display that was used to master the HDR content.
The HDR metadata is currently defined on the drm_connector via the hdr_output_metadata blob property.
It might be useful to define per-plane hdr metadata, as different planes might have been mastered differently.
I don't think this would directly help with the dynamic range blending problem. You still need to establish the mapping between the optical values from two different EOTFs and dynamic ranges. Or can you know which optical values match the mastering display maximum and minimum luminances for not-PQ?
## SDR Luminance
Since SDR covers a smaller luminance range than HDR, an SDR plane might look dark when blended with HDR content. Since the max HDR luminance can be quite variable (200-1,500 nits on actual displays) it is best to make the SDR maximum luminance value configurable.
**We propose a drm_plane property to specfy the desired maximum luminance of the SDR plane in nits.** This allows us to map the SDR content predictably into HDR's absolute luminance space.
What would be the mapping? Simple linear scaling? A more complicated tone mapping?
Rather than "SDR luminance", do you perhaps intend this to configure the dynamic range of the non-absolute-luminance EOTFs? In that case maybe you'd need a black luminance level too?
## Let There Be Color
So far we've only talked about luminance, ignoring colors altogether. Just like in the luminance space, traditionally the color space of display outputs has not been well defined. Similar to how an EOTF defines a mapping of pixel data to an absolute luminance value, the color space maps color information for each pixel onto the CIE 1931 chromaticity space. This can be thought of as a mapping to an absolute, real-life, color value.
A color space is defined by its primaries and white point. The primaries and white point are expressed as coordinates in the CIE 1931 color space. Think of the red primary as the reddest red that can be displayed within the color space. Same for green and blue.
Examples of color spaces are:
| Color Space | Description | | ----------- | ------------------------------------------ | | BT 601 | similar to BT 709 | | BT 709 | used by sRGB content; ~53% of BT 2020 | | DCI-P3 | used by most HDR displays; ~72% of BT 2020 | | BT 2020 | standard for most HDR content |
The color space is defined in DRM for YCbCr planes via the color_encoding property of the drm_plane.
I don't think that is quite right.
As far I understand, COLOR_ENCODING property controls which matrix is used to convert from YCbCr to RGB, but that is all it does. It is not used for the actual color space. So while these BT standards do specify the chromaticities, they also specify the YCbCr encoding which is the part used in this property.
YCbCr and RGB are color models. They are not color spaces. RGB is an additive color model while YCbCr is not, AFAIU. Blending requires an additive color model and linear luminance encoding.
You need two color space definitions to create one color space transformation: source color space and destination color space. You also need an idea *how* the two color spaces should be mapped, which is called "rendering intent". You can't do anything with just one color space definition, except to pass it on along with the pixels.
To be able to blend planes together, all planes need to be converted to the same color space first: the blending color space, whatever you choose it to be. I do not see where KMS would do this color space conversion, or where it would get the definition of the blending color space.
**We propose to add definitions for the RGB variants of the BT color spaces.**
Therefore I'm not sure this makes sense.
## Color Primaries and White Point
Just like displays can currently not represent the entire 0.0005 - 10,000 nits HDR range of the PQ 2084 EOTF, they are currently not capable of representing the entire BT.2020 color Gamut. For this reason video content will often specify the color primaries and white point used to master the video, in order to allow displays to be able to map the image as best as possible onto the display's gamut.
## Displays and Tonemapping
External displays are able to do their own tone and color mapping, based on the mastering luminance, color primaries, and white space defined in the HDR metadata.
Internal panels (which are currently few and far between) usually don't include the complex HW to do tone and color mapping on their own and will require the display driver to perform appropriate mapping.
FWIW, when designing Weston's color management, we are aiming for the latter "simple" panels foremost, because that gives us full control of all color conversions and tone mappings.
OTOH, if Weston has to present to a display which only accepts e.g. BT.2020/PQ signal, the display might always mangle the image in unexpected ways. Therefore I expect that by default Weston will do everything it can to try to make the display not apply anything magic image enhancement: trust that EDID description of the display gamut and dynamic range are correct, and use HDR metadata to tell the display that those values are exactly what we are using. And we use them.
IMO, a display doing its tone mapping magically is only useful when you want to be able to use "simple" playback devices that cannot adapt to the display they are driving. Magic tone mapping is also a way for hardware vendors to differentiate, which from the color management perspective is harmful as it makes it more difficult or impossible to predict the display behaviour or to keep it consistent.
So there are two opposing goals:
Traditional color management wants absolute control of the display, leaving nothing unpredictable and preferably also nothing undefined. Undefined behaviour can always be measured (profiled) which makes it predictable and known. The viewing environment is controlled and constant.
Entertainment wants the most visually impressive image quality by dynamically adapting to both displayed content and to the viewing environment conditions.
## Pixel Formats
The pixel formats, such as ARGB8888, ARGB2101010, P010, or FP16 are unrelated to color space and EOTF definitions. HDR pixels can be formatted in different ways but in order to not lose precision HDR content requires at least 10 bpc precision. For this reason ARGB2101010, P010, and FP16 are the obvious candidates for HDR. ARGB2101010 and P010 have the advantage of requiring only half the bandwidth as FP16, while FP16 has the advantage of enough precision to operate in a linear space, i.e. without EOTF.
Right.
## Proposed use-cases
Although the userspace side of this work is still in the early stages it is clear that we will want to support the following two use-cases:
**One XRGB2101010 HDR Plane:** A single, composited plane of HDR content. The use-case is a video player on a desktop with the compositor owning the composition of SDR and HDR content. The content shall be PQ BT.2020 formatted. The drm_connector's hdr_output_metadata shall be set.
This use case is already possible, right?
**One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc desktop plane, with a P010 HDR video plane underlayed. The HDR plane shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR boost value. The drm_connector's hdr_output_metadata shall be set.
This use case requires blending in KMS, so is the primary goal I suppose.
**One XRGB8888 SDR Plane - HDR output:** In order to support a smooth transition we recommend an OS that supports HDR output to provide the hdr_output_metadata on the drm_connector to configure the output for HDR, even when the content is only SDR. This will allow for a smooth transition between SDR-only and HDR content. In this use-case the SDR max luminance value should be provided on the drm_plane.
I think this might be already possible by crafting a CRTC GAMMA LUT? Not sure about precision.
In DCN we will de-PQ or de-Gamma all input in order to blend in linear space. For SDR content we will also apply any desired boost before blending. After blending we will then re-apply the PQ EOTF and do RGB to YCbCr conversion if needed.
This assumes the same color space over everything.
## Summary of proposed interface changes
per drm_plane:
- new RGB color space definitions, mirroring the existing YUV color
space definitions
- new transfer function property
- new SDR maximum white level property
How will these new KMS properties interact with per-plane DEGAMMA, CTM and/or GAMMA properties?
Why go with your proposal instead of per-plane CTM and LUT?
I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs both per-plane and on CRTC, would be:
plane: FB -> M1 -> LUT1 -> M2 -> blending input
CRTC: blending output -> LUT2 -> M3 -> connector
FB: framebuffer M1: matrix transform, capable of converting e.g. YCbCr to RGB LUT1: 1D LUT for content EOTF, to produce light-linear RGB M2: matrix transform for color space transformation
LUT2: 1D LUT for applying monitor EOTF^-1 M3: matrix transform, e.g. if you need to push YCbCr on the connector
We also need to know where and how clipping happens.
I think this scheme would allow implementing everything you want, and it would not be tied to rigid enumerations, and it won't have any magical conversions done under the hood as you would need to do to convert from one enum space to another. It leaves the render intent to be defined by the userspace compositor, rather than building a fixed policy in the kernel.
Userspace would be setting transformation operators, not color spaces, to the kernel, allowing the blending space to be chosen by userspace. In Weston we aim to choose then blending color space to be the same as the output color space, except in optical (linear) encoding. The output color space can be non-standard, e.g. measured with a display profiler equipment.
I would expect gamut mapping, dynamic range mapping and tone mapping to be places where most experimentation and innovation happens, so implementing them in the kernel with just few or no controllable parameters at this time seems like it could become useless fast.
Thanks, pq
## References
[1] https://en.wikipedia.org/wiki/High-dynamic-range_video#Perceptual_Quantizer
## Further Reading
https://gitlab.freedesktop.org/swick/wayland-protocols/-/blob/color/unstable... http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP309.pdf https://app.spectracal.com/Documents/White%20Papers/HDR_Demystified.pdf
Bhawanpreet Lakha (3): drm/color: Add RGB Color encodings drm/color: Add Color transfer functions for HDR/SDR drm/color: Add sdr boost property
.../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 4 +- .../gpu/drm/arm/display/komeda/komeda_plane.c | 4 +- drivers/gpu/drm/arm/malidp_planes.c | 4 +- drivers/gpu/drm/armada/armada_overlay.c | 4 +- drivers/gpu/drm/drm_atomic_uapi.c | 8 ++ drivers/gpu/drm/drm_color_mgmt.c | 84 +++++++++++++++++-- drivers/gpu/drm/i915/display/intel_sprite.c | 4 +- .../drm/i915/display/skl_universal_plane.c | 4 +- drivers/gpu/drm/nouveau/dispnv04/overlay.c | 4 +- drivers/gpu/drm/omapdrm/omap_plane.c | 4 +- drivers/gpu/drm/sun4i/sun8i_vi_layer.c | 4 +- drivers/gpu/drm/tidss/tidss_plane.c | 6 +- include/drm/drm_color_mgmt.h | 25 +++++- include/drm/drm_plane.h | 30 +++++++ 14 files changed, 173 insertions(+), 16 deletions(-)
On Wed, 28 Apr 2021 13:24:27 +0530 Shashank Sharma shashank.sharma@amd.com wrote:
Assuming these details, A compositor will look for DRM color properties like these:
Degamma plane property : To make buffers linear for Gamut mapping
Gamut mapping plane property: To gamut map SRGB buffer to BT2020 colorspace
Color space conversion plane property: To convert from YCBCR->RGB
Tone mapping plane property: To tone map SDR buffer S2H and HDR buffer H2H
Gamma plane/CRTC property: to re-apply the output ST2084 curve
...
* * * * ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌────────────────┐ * HDR 600 Nits│ │HDR 600 Nits │ │HDR600 │ │HDR500 │ │ HDR500 * ────────► │ Degamma ├────────────►│ Color space ├──────────►│ Tone mapping ├──────►│ Gamma │ * BT2020 │ OETF ST2084 │ BT2020 │ conversion │BT2020 │ H2H │BT2020 │ ST2084 │ BT2020 * YCBCR420 │ │ YCBCR420 │ YCBCR->RGB │RGB88 │ 600->500 │RGB888 │ │ RGB888 * Non Linear └─────────────────┘ Linear └─────────────────┘Linear └─────────────────┘Linear └────────────────┘ ST2084 */
Hi Shashank,
I think you might have degamma and color model conversion reversed, or is that a new thing in the HDR specs?
Usually the YCbCr/RGB conversion matrix applies to non-linear values AFAIU.
There is also confusion with OETF vs. EOTF. I got that initially wrong too. OETF is not just a name for inverse-EOTF but it is used in a different context. Though here it seems to be just a typo.
OETF is inherent to a camera when it converts light into electrical signals. EOTF is inherent to a monitor when it converts electrical signals to light. Depending on what the electrical signals have been defined to be in each step of a broadcasting chain, you might need OETF or EOTF or their inverse or a different OETF or EOTF or their inverse.
As we are talking about displays and likely assuming display-referred content (not scene-referred content), we probably have no use for OETF, but we could have several different EOTFs.
Thanks, pq
Hello Pekka,
On 30/04/21 15:13, Pekka Paalanen wrote:
On Wed, 28 Apr 2021 13:24:27 +0530 Shashank Sharma shashank.sharma@amd.com wrote:
Assuming these details, A compositor will look for DRM color properties like these:
Degamma plane property : To make buffers linear for Gamut mapping
Gamut mapping plane property: To gamut map SRGB buffer to BT2020 colorspace
Color space conversion plane property: To convert from YCBCR->RGB
Tone mapping plane property: To tone map SDR buffer S2H and HDR buffer H2H
Gamma plane/CRTC property: to re-apply the output ST2084 curve
...
* * * * ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌────────────────┐ * HDR 600 Nits│ │HDR 600 Nits │ │HDR600 │ │HDR500 │ │ HDR500 * ────────► │ Degamma ├────────────►│ Color space ├──────────►│ Tone mapping ├──────►│ Gamma │ * BT2020 │ OETF ST2084 │ BT2020 │ conversion │BT2020 │ H2H │BT2020 │ ST2084 │ BT2020 * YCBCR420 │ │ YCBCR420 │ YCBCR->RGB │RGB88 │ 600->500 │RGB888 │ │ RGB888 * Non Linear └─────────────────┘ Linear └─────────────────┘Linear └─────────────────┘Linear └────────────────┘ ST2084 */
Hi Shashank,
I think you might have degamma and color model conversion reversed, or is that a new thing in the HDR specs?
Usually the YCbCr/RGB conversion matrix applies to non-linear values AFAIU.
Ah, that was due to the Gamut mapping block. You are right, color format conversion can happen on non-linear data (doesn't mean it can't happen on linear), but in the sequential block above, there was gamut mapping (color space conversion), which needs to be done on Linear space, and I was a bit too lazy to create separate blocks, so I just re[placed the block titles :D.
There is also confusion with OETF vs. EOTF. I got that initially wrong too. OETF is not just a name for inverse-EOTF but it is used in a different context. Though here it seems to be just a typo. OETF is inherent to a camera when it converts light into electrical signals. EOTF is inherent to a monitor when it converts electrical signals to light. Depending on what the electrical signals have been defined to be in each step of a broadcasting chain, you might need OETF or EOTF or their inverse or a different OETF or EOTF or their inverse.
Yes, that was a typo. The intention was to call it inverse curve for HDR encoded buffers. It's almost 4 years (and 2 companies) since I last did HDR, so I am a bit rusty on the topic ;) .
- Shashank
As we are talking about displays and likely assuming display-referred content (not scene-referred content), we probably have no use for OETF, but we could have several different EOTFs.
Thanks, pq
On 2021-04-30 6:39 a.m., Shashank Sharma wrote:
Hello Pekka,
On 30/04/21 15:13, Pekka Paalanen wrote:
On Wed, 28 Apr 2021 13:24:27 +0530 Shashank Sharma shashank.sharma@amd.com wrote:
Assuming these details, A compositor will look for DRM color properties like these:
Degamma plane property : To make buffers linear for Gamut mapping
Gamut mapping plane property: To gamut map SRGB buffer to BT2020 colorspace
Color space conversion plane property: To convert from YCBCR->RGB
Tone mapping plane property: To tone map SDR buffer S2H and HDR buffer H2H
Gamma plane/CRTC property: to re-apply the output ST2084 curve
...
* * * * ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌────────────────┐ * HDR 600 Nits│ │HDR 600 Nits │ │HDR600 │ │HDR500 │ │ HDR500 * ────────► │ Degamma ├────────────►│ Color space ├──────────►│ Tone mapping ├──────►│ Gamma │ * BT2020 │ OETF ST2084 │ BT2020 │ conversion │BT2020 │ H2H │BT2020 │ ST2084 │ BT2020 * YCBCR420 │ │ YCBCR420 │ YCBCR->RGB │RGB88 │ 600->500 │RGB888 │ │ RGB888 * Non Linear └─────────────────┘ Linear └─────────────────┘Linear └─────────────────┘Linear └────────────────┘ ST2084 */
Hi Shashank,
I think you might have degamma and color model conversion reversed, or is that a new thing in the HDR specs?
Usually the YCbCr/RGB conversion matrix applies to non-linear values AFAIU.
Ah, that was due to the Gamut mapping block. You are right, color format conversion can happen on non-linear data (doesn't mean it can't happen on linear), but in the sequential block above, there was gamut mapping (color space conversion), which needs to be done on Linear space, and I was a bit too lazy to create separate blocks, so I just re[placed the block titles :D.
There is also confusion with OETF vs. EOTF. I got that initially wrong too. OETF is not just a name for inverse-EOTF but it is used in a different context. Though here it seems to be just a typo. OETF is inherent to a camera when it converts light into electrical signals. EOTF is inherent to a monitor when it converts electrical signals to light. Depending on what the electrical signals have been defined to be in each step of a broadcasting chain, you might need OETF or EOTF or their inverse or a different OETF or EOTF or their inverse.
Yes, that was a typo. The intention was to call it inverse curve for HDR encoded buffers. It's almost 4 years (and 2 companies) since I last did HDR, so I am a bit rusty on the topic ;) .
- Shashank
Thanks, Ville and Shashank. This is indeed helpful. I apologize for the late response but I needed to take some time to do more reading and internalize some of the HDR and CM concepts. I will send out a v2 of my patchset but realize that it is only a small step toward the right KMS interface for HDR and CM.
Harry
As we are talking about displays and likely assuming display-referred content (not scene-referred content), we probably have no use for OETF, but we could have several different EOTFs.
Thanks, pq
On 2021-04-27 10:50 a.m., Pekka Paalanen wrote:
On Mon, 26 Apr 2021 13:38:49 -0400 Harry Wentland harry.wentland@amd.com wrote:
## Introduction
We are looking to enable HDR support for a couple of single-plane and multi-plane scenarios. To do this effectively we recommend new interfaces to drm_plane. Below I'll give a bit of background on HDR and why we propose these interfaces.
## Defining a pixel's luminance
Currently the luminance space of pixels in a framebuffer/plane presented to the display is not well defined. It's usually assumed to be in a 2.2 or 2.4 gamma space and has no mapping to an absolute luminance value but is interpreted in relative terms.
Luminance can be measured and described in absolute terms as candela per meter squared, or cd/m2, or nits. Even though a pixel value can be mapped to luminance in a linear fashion to do so without losing a lot of detail requires 16-bpc color depth. The reason for this is that human perception can distinguish roughly between a 0.5-1% luminance delta. A linear representation is suboptimal, wasting precision in the highlights and losing precision in the shadows.
A gamma curve is a decent approximation to a human's perception of luminance, but the PQ (perceptual quantizer) function [1] improves on it. It also defines the luminance values in absolute terms, with the highest value being 10,000 nits and the lowest 0.0005 nits.
Using a content that's defined in PQ space we can approximate the real world in a much better way.
Here are some examples of real-life objects and their approximate luminance values:
| Object | Luminance in nits | | ----------------- | ----------------- | | Sun | 1.6 million | | Fluorescent light | 10,000 | | Highlights | 1,000 - sunlight | | White Objects | 250 - 1,000 | | Typical objects | 1 - 250 | | Shadows | 0.01 - 1 | | Ultra Blacks | 0 - 0.0005 |
## Describing the luminance space
**We propose a new drm_plane property to describe the Eletro-Optical Transfer Function (EOTF) with which its framebuffer was composed.** Examples of EOTF are:
| EOTF | Description | | --------- |:------------------------------------------------------------------------- | | Gamma 2.2 | a simple 2.2 gamma | | sRGB | 2.4 gamma with small initial linear section | | PQ 2084 | SMPTE ST 2084; used for HDR video and allows for up to 10,000 nit support | | Linear | Linear relationship between pixel value and luminance value |
The definitions agree with what I have learnt so far. However, with these EOTF definitions, only PQ defines absolute luminance values while the others do not. So this is not enough information to blend planes together if they do not all use the same EOTF with the same dynamic range. More below.
Good point.
## Mastering Luminances
Now we are able to use the PQ 2084 EOTF to define the luminance of pixels in absolute terms. Unfortunately we're again presented with physical limitations of the display technologies on the market today. Here are a few examples of luminance ranges of displays.
| Display | Luminance range in nits | | ------------------------ | ----------------------- | | Typical PC display | 0.3 - 200 | | Excellent LCD HDTV | 0.3 - 400 | | HDR LCD w/ local dimming | 0.05 - 1,500 |
Since no display can currently show the full 0.0005 to 10,000 nits luminance range the display will need to tonemap the HDR content, i.e to fit the content within a display's capabilities. To assist with tonemapping HDR content is usually accompanied with a metadata that describes (among other things) the minimum and maximum mastering luminance, i.e. the maximum and minimum luminance of the display that was used to master the HDR content.
The HDR metadata is currently defined on the drm_connector via the hdr_output_metadata blob property.
It might be useful to define per-plane hdr metadata, as different planes might have been mastered differently.
I don't think this would directly help with the dynamic range blending problem. You still need to establish the mapping between the optical values from two different EOTFs and dynamic ranges. Or can you know which optical values match the mastering display maximum and minimum luminances for not-PQ?
My understanding of this is probably best illustrated by this example:
Assume HDR was mastered on a display with a maximum white level of 500 nits and played back on a display that supports a max white level of 400 nits. If you know the mastering white level of 500 you know that this is the maximum value you need to compress down to 400 nits, allowing you to use the full extent of the 400 nits panel.
If you do not know the mastering luminance is 500 nits you would have to compress 10,000 nits down to 400 (assuming PQ), losing quite a bit of the full 400 nits available as you'll need room to map the 500 to 10,000 nits range which in reality is completely unused. You might end up with mapping 500 nits to 350 nits, instead of mapping it to 400.
## SDR Luminance
Since SDR covers a smaller luminance range than HDR, an SDR plane might look dark when blended with HDR content. Since the max HDR luminance can be quite variable (200-1,500 nits on actual displays) it is best to make the SDR maximum luminance value configurable.
**We propose a drm_plane property to specfy the desired maximum luminance of the SDR plane in nits.** This allows us to map the SDR content predictably into HDR's absolute luminance space.
What would be the mapping? Simple linear scaling? A more complicated tone mapping?
Rather than "SDR luminance", do you perhaps intend this to configure the dynamic range of the non-absolute-luminance EOTFs? In that case maybe you'd need a black luminance level too?
SDR luminance is essentially how AMD HW lets us deal with this, but conceptually you're right, this could extend beyond SDR to any non-absolute-luminance (NAL?) EOTF.
I don't think AMD HW currently has a way to program a black luminance level unless we turn the white/black luminance into a LUT.
## Let There Be Color
So far we've only talked about luminance, ignoring colors altogether. Just like in the luminance space, traditionally the color space of display outputs has not been well defined. Similar to how an EOTF defines a mapping of pixel data to an absolute luminance value, the color space maps color information for each pixel onto the CIE 1931 chromaticity space. This can be thought of as a mapping to an absolute, real-life, color value.
A color space is defined by its primaries and white point. The primaries and white point are expressed as coordinates in the CIE 1931 color space. Think of the red primary as the reddest red that can be displayed within the color space. Same for green and blue.
Examples of color spaces are:
| Color Space | Description | | ----------- | ------------------------------------------ | | BT 601 | similar to BT 709 | | BT 709 | used by sRGB content; ~53% of BT 2020 | | DCI-P3 | used by most HDR displays; ~72% of BT 2020 | | BT 2020 | standard for most HDR content |
The color space is defined in DRM for YCbCr planes via the color_encoding property of the drm_plane.
I don't think that is quite right.
As far I understand, COLOR_ENCODING property controls which matrix is used to convert from YCbCr to RGB, but that is all it does. It is not used for the actual color space. So while these BT standards do specify the chromaticities, they also specify the YCbCr encoding which is the part used in this property.
YCbCr and RGB are color models. They are not color spaces. RGB is an additive color model while YCbCr is not, AFAIU. Blending requires an additive color model and linear luminance encoding.
You need two color space definitions to create one color space transformation: source color space and destination color space. You also need an idea *how* the two color spaces should be mapped, which is called "rendering intent". You can't do anything with just one color space definition, except to pass it on along with the pixels.
To be able to blend planes together, all planes need to be converted to the same color space first: the blending color space, whatever you choose it to be. I do not see where KMS would do this color space conversion, or where it would get the definition of the blending color space.
**We propose to add definitions for the RGB variants of the BT color spaces.**
Therefore I'm not sure this makes sense.
You and Ville are absolutely correct and a reading of https://en.wikipedia.org/wiki/YCbCr set me straight. :)
## Color Primaries and White Point
Just like displays can currently not represent the entire 0.0005 - 10,000 nits HDR range of the PQ 2084 EOTF, they are currently not capable of representing the entire BT.2020 color Gamut. For this reason video content will often specify the color primaries and white point used to master the video, in order to allow displays to be able to map the image as best as possible onto the display's gamut.
## Displays and Tonemapping
External displays are able to do their own tone and color mapping, based on the mastering luminance, color primaries, and white space defined in the HDR metadata.
Internal panels (which are currently few and far between) usually don't include the complex HW to do tone and color mapping on their own and will require the display driver to perform appropriate mapping.
FWIW, when designing Weston's color management, we are aiming for the latter "simple" panels foremost, because that gives us full control of all color conversions and tone mappings.
OTOH, if Weston has to present to a display which only accepts e.g. BT.2020/PQ signal, the display might always mangle the image in unexpected ways. Therefore I expect that by default Weston will do everything it can to try to make the display not apply anything magic image enhancement: trust that EDID description of the display gamut and dynamic range are correct, and use HDR metadata to tell the display that those values are exactly what we are using. And we use them.
IMO, a display doing its tone mapping magically is only useful when you want to be able to use "simple" playback devices that cannot adapt to the display they are driving. Magic tone mapping is also a way for hardware vendors to differentiate, which from the color management perspective is harmful as it makes it more difficult or impossible to predict the display behaviour or to keep it consistent.
So there are two opposing goals:
Traditional color management wants absolute control of the display, leaving nothing unpredictable and preferably also nothing undefined. Undefined behaviour can always be measured (profiled) which makes it predictable and known. The viewing environment is controlled and constant.
Entertainment wants the most visually impressive image quality by dynamically adapting to both displayed content and to the viewing environment conditions.
I feel much of the mess around HDR and CM is all the magic that everyone tries to do to differentiate themselves. I'm not a fan of that.
One thing to watch out for is that the definitions for some of these things (I'm thinking PWL LUTs, 3D LUT, SDR boots) might differ a fair bit between HW vendors and we'd want to find a way to hide these complexities to KMS clients.
## Pixel Formats
The pixel formats, such as ARGB8888, ARGB2101010, P010, or FP16 are unrelated to color space and EOTF definitions. HDR pixels can be formatted in different ways but in order to not lose precision HDR content requires at least 10 bpc precision. For this reason ARGB2101010, P010, and FP16 are the obvious candidates for HDR. ARGB2101010 and P010 have the advantage of requiring only half the bandwidth as FP16, while FP16 has the advantage of enough precision to operate in a linear space, i.e. without EOTF.
Right.
## Proposed use-cases
Although the userspace side of this work is still in the early stages it is clear that we will want to support the following two use-cases:
**One XRGB2101010 HDR Plane:** A single, composited plane of HDR content. The use-case is a video player on a desktop with the compositor owning the composition of SDR and HDR content. The content shall be PQ BT.2020 formatted. The drm_connector's hdr_output_metadata shall be set.
This use case is already possible, right?
Yes, not sure how well tested it is but the AMD driver at least is programming the LUTs into bypass if we don't have a degamma or regamma LUT.
**One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc desktop plane, with a P010 HDR video plane underlayed. The HDR plane shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR boost value. The drm_connector's hdr_output_metadata shall be set.
This use case requires blending in KMS, so is the primary goal I suppose.
Correct, my understanding is that the key here is to ensure blending in KMS is done in same blending space (linear RGB) as blending using GPU shaders and SDR boost is applied in the same fashion.
**One XRGB8888 SDR Plane - HDR output:** In order to support a smooth transition we recommend an OS that supports HDR output to provide the hdr_output_metadata on the drm_connector to configure the output for HDR, even when the content is only SDR. This will allow for a smooth transition between SDR-only and HDR content. In this use-case the SDR max luminance value should be provided on the drm_plane.
I think this might be already possible by crafting a CRTC GAMMA LUT? Not sure about precision.
In DCN we will de-PQ or de-Gamma all input in order to blend in linear space. For SDR content we will also apply any desired boost before blending. After blending we will then re-apply the PQ EOTF and do RGB to YCbCr conversion if needed.
This assumes the same color space over everything.
## Summary of proposed interface changes
per drm_plane:
- new RGB color space definitions, mirroring the existing YUV color
space definitions
- new transfer function property
- new SDR maximum white level property
How will these new KMS properties interact with per-plane DEGAMMA, CTM and/or GAMMA properties?
Why go with your proposal instead of per-plane CTM and LUT?
I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs both per-plane and on CRTC, would be:
plane: FB -> M1 -> LUT1 -> M2 -> blending input
CRTC: blending output -> LUT2 -> M3 -> connector
FB: framebuffer M1: matrix transform, capable of converting e.g. YCbCr to RGB LUT1: 1D LUT for content EOTF, to produce light-linear RGB M2: matrix transform for color space transformation
LUT2: 1D LUT for applying monitor EOTF^-1 M3: matrix transform, e.g. if you need to push YCbCr on the connector
We also need to know where and how clipping happens.
I think this scheme would allow implementing everything you want, and it would not be tied to rigid enumerations, and it won't have any magical conversions done under the hood as you would need to do to convert from one enum space to another. It leaves the render intent to be defined by the userspace compositor, rather than building a fixed policy in the kernel.
Userspace would be setting transformation operators, not color spaces, to the kernel, allowing the blending space to be chosen by userspace. In Weston we aim to choose then blending color space to be the same as the output color space, except in optical (linear) encoding. The output color space can be non-standard, e.g. measured with a display profiler equipment.
I would expect gamut mapping, dynamic range mapping and tone mapping to be places where most experimentation and innovation happens, so implementing them in the kernel with just few or no controllable parameters at this time seems like it could become useless fast.
How important from a Wayland compositor perspective would it be to have fully configurable LUTs per plane, as opposed to providing KMS with a named transfer function, such as PQ or BT709, for LUT1 and LUT2?
M1 on AMD's driver is currently hidden in our driver and essentially always set to BT709 at the moment, though we should look at color_encoding to pick the right M1. Are the color_encoding and color_range properties enough for what you'd like to achieve for M1?
M3, currently the only matrix exposed by KMS (on drm_crtc) currently seems to be applied at M2 by AMD's driver. I need to dig into this a bit more. It would make sense to expose M2 and M3 as part of KMS (on drm_crtc and drm_plane).
I did a bunch of reading and thinking since your comments and created a v2 of the patchset. I don't think it's where we need it to be but I hope it's a step in the right direction. I will send it shortly.
Harry
Thanks, pq
## References
[1] https://en.wikipedia.org/wiki/High-dynamic-range_video#Perceptual_Quantizer
## Further Reading
https://gitlab.freedesktop.org/swick/wayland-protocols/-/blob/color/unstable... http://downloads.bbc.co.uk/rd/pubs/whp/whp-pdf-files/WHP309.pdf https://app.spectracal.com/Documents/White%20Papers/HDR_Demystified.pdf
Bhawanpreet Lakha (3): drm/color: Add RGB Color encodings drm/color: Add Color transfer functions for HDR/SDR drm/color: Add sdr boost property
.../gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 4 +- .../gpu/drm/arm/display/komeda/komeda_plane.c | 4 +- drivers/gpu/drm/arm/malidp_planes.c | 4 +- drivers/gpu/drm/armada/armada_overlay.c | 4 +- drivers/gpu/drm/drm_atomic_uapi.c | 8 ++ drivers/gpu/drm/drm_color_mgmt.c | 84 +++++++++++++++++-- drivers/gpu/drm/i915/display/intel_sprite.c | 4 +- .../drm/i915/display/skl_universal_plane.c | 4 +- drivers/gpu/drm/nouveau/dispnv04/overlay.c | 4 +- drivers/gpu/drm/omapdrm/omap_plane.c | 4 +- drivers/gpu/drm/sun4i/sun8i_vi_layer.c | 4 +- drivers/gpu/drm/tidss/tidss_plane.c | 6 +- include/drm/drm_color_mgmt.h | 25 +++++- include/drm/drm_plane.h | 30 +++++++ 14 files changed, 173 insertions(+), 16 deletions(-)
On Fri, 14 May 2021 17:05:11 -0400 Harry Wentland harry.wentland@amd.com wrote:
On 2021-04-27 10:50 a.m., Pekka Paalanen wrote:
On Mon, 26 Apr 2021 13:38:49 -0400 Harry Wentland harry.wentland@amd.com wrote:
...
## Mastering Luminances
Now we are able to use the PQ 2084 EOTF to define the luminance of pixels in absolute terms. Unfortunately we're again presented with physical limitations of the display technologies on the market today. Here are a few examples of luminance ranges of displays.
| Display | Luminance range in nits | | ------------------------ | ----------------------- | | Typical PC display | 0.3 - 200 | | Excellent LCD HDTV | 0.3 - 400 | | HDR LCD w/ local dimming | 0.05 - 1,500 |
Since no display can currently show the full 0.0005 to 10,000 nits luminance range the display will need to tonemap the HDR content, i.e to fit the content within a display's capabilities. To assist with tonemapping HDR content is usually accompanied with a metadata that describes (among other things) the minimum and maximum mastering luminance, i.e. the maximum and minimum luminance of the display that was used to master the HDR content.
The HDR metadata is currently defined on the drm_connector via the hdr_output_metadata blob property.
It might be useful to define per-plane hdr metadata, as different planes might have been mastered differently.
I don't think this would directly help with the dynamic range blending problem. You still need to establish the mapping between the optical values from two different EOTFs and dynamic ranges. Or can you know which optical values match the mastering display maximum and minimum luminances for not-PQ?
My understanding of this is probably best illustrated by this example:
Assume HDR was mastered on a display with a maximum white level of 500 nits and played back on a display that supports a max white level of 400 nits. If you know the mastering white level of 500 you know that this is the maximum value you need to compress down to 400 nits, allowing you to use the full extent of the 400 nits panel.
Right, but in the kernel, where do you get these nits values from?
hdr_output_metadata blob is infoframe data to the monitor. I think this should be independent of the metadata used for color transformations in the display pipeline before the monitor.
EDID may tell us the monitor HDR metadata, but again what is used in the color transformations should be independent, because EDIDs lie, lighting environments change, and users have different preferences.
What about black levels?
Do you want to do black level adjustment?
How exactly should the compression work?
Where do you map the mid-tones?
What if the end user wants something different?
If you do not know the mastering luminance is 500 nits you would have to compress 10,000 nits down to 400 (assuming PQ), losing quite a bit of the full 400 nits available as you'll need room to map the 500 to 10,000 nits range which in reality is completely unused. You might end up with mapping 500 nits to 350 nits, instead of mapping it to 400.
The quality of the result depends on the compression (tone-mapping) algorithm. I believe no-one will ever do a simple linear compression of ranges.
Instead, you probably do something smooth in the black end, keep mid-tones roughly as they are, and the again do a smooth saturation to some "reasonable" level that goes well with the monitor in the current lighting environment without introducing coloring artifacts, and just clip the huge overshoot of the full PQ-range.
There are many big and small decisions to be made in how to map out-of-gamut or out-of-brightness values into the displayable range, and no algorithm fits all use cases. I believe this is why e.g. ICC has several different "render intents", some of which are so vaguely defined that they are practically undefined - just like what "a good picture" means. You have essentially three dimensions: luminance, hue, and saturation. Which one will you sacrifice, shift or emphasize and to what degree to fit the square content peg into the round display hole?
A naive example: Let's say content has 300 nits red. Your display can show max 400 nits white or max 180 nits red, and anything in between. What will you show?
The problem is, that if UAPI does not define exactly what happens, then taking advantage of these hardware capabilities means you have no idea what happens to your content. This may be fine for closed systems, where the software has been carefully built for the specific hardware revision and the use cases of the complete system have been pre-defined, so that it can assume what should and will happen. But should that be exposed as a generic UAPI, when generic userspace has no chance of knowing what it will do?
...
## Displays and Tonemapping
External displays are able to do their own tone and color mapping, based on the mastering luminance, color primaries, and white space defined in the HDR metadata.
Internal panels (which are currently few and far between) usually don't include the complex HW to do tone and color mapping on their own and will require the display driver to perform appropriate mapping.
FWIW, when designing Weston's color management, we are aiming for the latter "simple" panels foremost, because that gives us full control of all color conversions and tone mappings.
OTOH, if Weston has to present to a display which only accepts e.g. BT.2020/PQ signal, the display might always mangle the image in unexpected ways. Therefore I expect that by default Weston will do everything it can to try to make the display not apply anything magic image enhancement: trust that EDID description of the display gamut and dynamic range are correct, and use HDR metadata to tell the display that those values are exactly what we are using. And we use them.
IMO, a display doing its tone mapping magically is only useful when you want to be able to use "simple" playback devices that cannot adapt to the display they are driving. Magic tone mapping is also a way for hardware vendors to differentiate, which from the color management perspective is harmful as it makes it more difficult or impossible to predict the display behaviour or to keep it consistent.
So there are two opposing goals:
Traditional color management wants absolute control of the display, leaving nothing unpredictable and preferably also nothing undefined. Undefined behaviour can always be measured (profiled) which makes it predictable and known. The viewing environment is controlled and constant.
Entertainment wants the most visually impressive image quality by dynamically adapting to both displayed content and to the viewing environment conditions.
I feel much of the mess around HDR and CM is all the magic that everyone tries to do to differentiate themselves. I'm not a fan of that.
One thing to watch out for is that the definitions for some of these things (I'm thinking PWL LUTs, 3D LUT, SDR boots) might differ a fair bit between HW vendors and we'd want to find a way to hide these complexities to KMS clients.
How could we hide them?
Actually, what do you mean with "hide"?
...
**One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc desktop plane, with a P010 HDR video plane underlayed. The HDR plane shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR boost value. The drm_connector's hdr_output_metadata shall be set.
This use case requires blending in KMS, so is the primary goal I suppose.
Correct, my understanding is that the key here is to ensure blending in KMS is done in same blending space (linear RGB) as blending using GPU shaders and SDR boost is applied in the same fashion.
Yes, but this is also something I think cannot be achieved by telling the kernel about color spaces and letting the driver come up with a color transformation. I believe this can only be achieved by telling the kernel the color transformation directly, and also describing to userspace what kind of color transformations are possible (1D LUT, 3D LUT, other; precision...).
...
## Summary of proposed interface changes
per drm_plane:
- new RGB color space definitions, mirroring the existing YUV color
space definitions
- new transfer function property
- new SDR maximum white level property
How will these new KMS properties interact with per-plane DEGAMMA, CTM and/or GAMMA properties?
Why go with your proposal instead of per-plane CTM and LUT?
I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs both per-plane and on CRTC, would be:
plane: FB -> M1 -> LUT1 -> M2 -> blending input
CRTC: blending output -> LUT2 -> M3 -> connector
FB: framebuffer M1: matrix transform, capable of converting e.g. YCbCr to RGB LUT1: 1D LUT for content EOTF, to produce light-linear RGB M2: matrix transform for color space transformation
LUT2: 1D LUT for applying monitor EOTF^-1 M3: matrix transform, e.g. if you need to push YCbCr on the connector
We also need to know where and how clipping happens.
I think this scheme would allow implementing everything you want, and it would not be tied to rigid enumerations, and it won't have any magical conversions done under the hood as you would need to do to convert from one enum space to another. It leaves the render intent to be defined by the userspace compositor, rather than building a fixed policy in the kernel.
Userspace would be setting transformation operators, not color spaces, to the kernel, allowing the blending space to be chosen by userspace. In Weston we aim to choose then blending color space to be the same as the output color space, except in optical (linear) encoding. The output color space can be non-standard, e.g. measured with a display profiler equipment.
I would expect gamut mapping, dynamic range mapping and tone mapping to be places where most experimentation and innovation happens, so implementing them in the kernel with just few or no controllable parameters at this time seems like it could become useless fast.
How important from a Wayland compositor perspective would it be to have fully configurable LUTs per plane, as opposed to providing KMS with a named transfer function, such as PQ or BT709, for LUT1 and LUT2?
For Wayland desktop compositors, absolutely essential in my opinion.
I would strongly advice against any desktop compositor from using the "named transfer function" API and instead do everything in GPU shaders if no other KMS way exists, unless the KMS UAPI definition gives the exact mathematical formula of what that step will do.
For entertainment devices using Wayland as an internal detail, who cares. The system manufacturer makes things work any way they want, and you cannot run non-curated apps on those devices, so there are no compatibility issues.
Then there would be systems between those two extremes, which would likely use a mixture of those approaches, e.g. Kodi on bare hardware I imagine (no Wayland there, but Wayland is kind of unrelated except for the ideology).
Mind, I'm not talking only about EOTFs here, but thinking of the whole color management. EOTFs per se are simple. Tone-mapping, including gamut mapping and luminance mapping, are hard because there is no one correct formula.
However, if we are talking about *only* EOTFs, then an enumeration of the curves could be usable. But you were also talking about configuring dynamic ranges and doing luminance range mapping in the driver, which I think won't work with enums, but could perhaps be accounted for in M2 by userspace. Then again, if you have enums and not a LUT, you cannot implement custom non-linear per-channel mapping which might be useful for e.g. adapting SDR content for a HDR blending space. Or HDR content to a different HDR blending space.
See, a LUT can be more than just EOTF: it can apply a part of the tone-mapping too. An enum cannot.
I have a feeling that most tone-mapping curves would be non-linear with some sort of S-shape and clipping of black and white ends. But that is also a naive view, because tone-mapping is not a 1D-curve, it's a mapping from R^3 to R^3 in RGB since on a 600 nit monitor you cannot do 600 nit red for instance. On a 600 nit monitor you can do exactly one color with 600 nits: white.
M1 on AMD's driver is currently hidden in our driver and essentially always set to BT709 at the moment, though we should look at color_encoding to pick the right M1. Are the color_encoding and color_range properties enough for what you'd like to achieve for M1?
Probably yes, for standard format content, until a new standard comes up again and then you have to add another enum value in the UAPI. M1 is probably the best place where an enum fits in the pipeline though, and for M3 as well because non-RGB inputs to monitors are quite standardised, and at those steps I would assume there are no parts of tone-mapping done.
M3, currently the only matrix exposed by KMS (on drm_crtc) currently seems to be applied at M2 by AMD's driver. I need to dig into this a bit more. It would make sense to expose M2 and M3 as part of KMS (on drm_crtc and drm_plane).
Good, it's really important to make sure that all drivers agree on the pipeline layout that is defined(???) in the KMS UAPI specification.
I did a bunch of reading and thinking since your comments and created a v2 of the patchset. I don't think it's where we need it to be but I hope it's a step in the right direction. I will send it shortly.
I have to say I'm still very sceptical about this enum-based approach as you have noticed. :-)
Thanks, pq
On 2021-05-17 10:57, Pekka Paalanen wrote:
On Fri, 14 May 2021 17:05:11 -0400 Harry Wentland harry.wentland@amd.com wrote:
On 2021-04-27 10:50 a.m., Pekka Paalanen wrote:
On Mon, 26 Apr 2021 13:38:49 -0400 Harry Wentland harry.wentland@amd.com wrote:
...
## Mastering Luminances
Now we are able to use the PQ 2084 EOTF to define the luminance of pixels in absolute terms. Unfortunately we're again presented with physical limitations of the display technologies on the market today. Here are a few examples of luminance ranges of displays.
| Display | Luminance range in nits | | ------------------------ | ----------------------- | | Typical PC display | 0.3 - 200 | | Excellent LCD HDTV | 0.3 - 400 | | HDR LCD w/ local dimming | 0.05 - 1,500 |
Since no display can currently show the full 0.0005 to 10,000 nits luminance range the display will need to tonemap the HDR content, i.e to fit the content within a display's capabilities. To assist with tonemapping HDR content is usually accompanied with a metadata that describes (among other things) the minimum and maximum mastering luminance, i.e. the maximum and minimum luminance of the display that was used to master the HDR content.
The HDR metadata is currently defined on the drm_connector via the hdr_output_metadata blob property.
It might be useful to define per-plane hdr metadata, as different planes might have been mastered differently.
I don't think this would directly help with the dynamic range blending problem. You still need to establish the mapping between the optical values from two different EOTFs and dynamic ranges. Or can you know which optical values match the mastering display maximum and minimum luminances for not-PQ?
My understanding of this is probably best illustrated by this example:
Assume HDR was mastered on a display with a maximum white level of 500 nits and played back on a display that supports a max white level of 400 nits. If you know the mastering white level of 500 you know that this is the maximum value you need to compress down to 400 nits, allowing you to use the full extent of the 400 nits panel.
Right, but in the kernel, where do you get these nits values from?
hdr_output_metadata blob is infoframe data to the monitor. I think this should be independent of the metadata used for color transformations in the display pipeline before the monitor.
EDID may tell us the monitor HDR metadata, but again what is used in the color transformations should be independent, because EDIDs lie, lighting environments change, and users have different preferences.
What about black levels?
Do you want to do black level adjustment?
How exactly should the compression work?
Where do you map the mid-tones?
What if the end user wants something different?
I suspect that this is not about tone mapping at all. The use cases listed always have the display in PQ mode and just assume that no content exceeds the PQ limitations. Then you can simply bring all content to the color space with a matrix multiplication and then map the linear light content somewhere into the PQ range. Tone mapping is performed in the display only.
From a generic wayland compositor point of view this is uninteresting.
I completely agree with what you said below though. I would even argue that all generic KMS abstract pipeline elements must have a well defined place in the pipeline and follow an exact specified formula.
If you do not know the mastering luminance is 500 nits you would have to compress 10,000 nits down to 400 (assuming PQ), losing quite a bit of the full 400 nits available as you'll need room to map the 500 to 10,000 nits range which in reality is completely unused. You might end up with mapping 500 nits to 350 nits, instead of mapping it to 400.
The quality of the result depends on the compression (tone-mapping) algorithm. I believe no-one will ever do a simple linear compression of ranges.
Instead, you probably do something smooth in the black end, keep mid-tones roughly as they are, and the again do a smooth saturation to some "reasonable" level that goes well with the monitor in the current lighting environment without introducing coloring artifacts, and just clip the huge overshoot of the full PQ-range.
There are many big and small decisions to be made in how to map out-of-gamut or out-of-brightness values into the displayable range, and no algorithm fits all use cases. I believe this is why e.g. ICC has several different "render intents", some of which are so vaguely defined that they are practically undefined - just like what "a good picture" means. You have essentially three dimensions: luminance, hue, and saturation. Which one will you sacrifice, shift or emphasize and to what degree to fit the square content peg into the round display hole?
A naive example: Let's say content has 300 nits red. Your display can show max 400 nits white or max 180 nits red, and anything in between. What will you show?
The problem is, that if UAPI does not define exactly what happens, then taking advantage of these hardware capabilities means you have no idea what happens to your content. This may be fine for closed systems, where the software has been carefully built for the specific hardware revision and the use cases of the complete system have been pre-defined, so that it can assume what should and will happen. But should that be exposed as a generic UAPI, when generic userspace has no chance of knowing what it will do?
...
## Displays and Tonemapping
External displays are able to do their own tone and color mapping, based on the mastering luminance, color primaries, and white space defined in the HDR metadata.
Internal panels (which are currently few and far between) usually don't include the complex HW to do tone and color mapping on their own and will require the display driver to perform appropriate mapping.
FWIW, when designing Weston's color management, we are aiming for the latter "simple" panels foremost, because that gives us full control of all color conversions and tone mappings.
OTOH, if Weston has to present to a display which only accepts e.g. BT.2020/PQ signal, the display might always mangle the image in unexpected ways. Therefore I expect that by default Weston will do everything it can to try to make the display not apply anything magic image enhancement: trust that EDID description of the display gamut and dynamic range are correct, and use HDR metadata to tell the display that those values are exactly what we are using. And we use them.
IMO, a display doing its tone mapping magically is only useful when you want to be able to use "simple" playback devices that cannot adapt to the display they are driving. Magic tone mapping is also a way for hardware vendors to differentiate, which from the color management perspective is harmful as it makes it more difficult or impossible to predict the display behaviour or to keep it consistent.
So there are two opposing goals:
Traditional color management wants absolute control of the display, leaving nothing unpredictable and preferably also nothing undefined. Undefined behaviour can always be measured (profiled) which makes it predictable and known. The viewing environment is controlled and constant.
Entertainment wants the most visually impressive image quality by dynamically adapting to both displayed content and to the viewing environment conditions.
I feel much of the mess around HDR and CM is all the magic that everyone tries to do to differentiate themselves. I'm not a fan of that.
One thing to watch out for is that the definitions for some of these things (I'm thinking PWL LUTs, 3D LUT, SDR boots) might differ a fair bit between HW vendors and we'd want to find a way to hide these complexities to KMS clients.
How could we hide them?
Actually, what do you mean with "hide"?
...
**One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc desktop plane, with a P010 HDR video plane underlayed. The HDR plane shall be PQ BT.2020 formatted. The desktop plane shall specify an SDR boost value. The drm_connector's hdr_output_metadata shall be set.
This use case requires blending in KMS, so is the primary goal I suppose.
Correct, my understanding is that the key here is to ensure blending in KMS is done in same blending space (linear RGB) as blending using GPU shaders and SDR boost is applied in the same fashion.
Yes, but this is also something I think cannot be achieved by telling the kernel about color spaces and letting the driver come up with a color transformation. I believe this can only be achieved by telling the kernel the color transformation directly, and also describing to userspace what kind of color transformations are possible (1D LUT, 3D LUT, other; precision...).
...
## Summary of proposed interface changes
per drm_plane:
- new RGB color space definitions, mirroring the existing YUV color
space definitions
- new transfer function property
- new SDR maximum white level property
How will these new KMS properties interact with per-plane DEGAMMA, CTM and/or GAMMA properties?
Why go with your proposal instead of per-plane CTM and LUT?
I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs both per-plane and on CRTC, would be:
plane: FB -> M1 -> LUT1 -> M2 -> blending input
CRTC: blending output -> LUT2 -> M3 -> connector
FB: framebuffer M1: matrix transform, capable of converting e.g. YCbCr to RGB LUT1: 1D LUT for content EOTF, to produce light-linear RGB M2: matrix transform for color space transformation
LUT2: 1D LUT for applying monitor EOTF^-1 M3: matrix transform, e.g. if you need to push YCbCr on the connector
We also need to know where and how clipping happens.
I think this scheme would allow implementing everything you want, and it would not be tied to rigid enumerations, and it won't have any magical conversions done under the hood as you would need to do to convert from one enum space to another. It leaves the render intent to be defined by the userspace compositor, rather than building a fixed policy in the kernel.
Userspace would be setting transformation operators, not color spaces, to the kernel, allowing the blending space to be chosen by userspace. In Weston we aim to choose then blending color space to be the same as the output color space, except in optical (linear) encoding. The output color space can be non-standard, e.g. measured with a display profiler equipment.
I would expect gamut mapping, dynamic range mapping and tone mapping to be places where most experimentation and innovation happens, so implementing them in the kernel with just few or no controllable parameters at this time seems like it could become useless fast.
How important from a Wayland compositor perspective would it be to have fully configurable LUTs per plane, as opposed to providing KMS with a named transfer function, such as PQ or BT709, for LUT1 and LUT2?
For Wayland desktop compositors, absolutely essential in my opinion.
I would strongly advice against any desktop compositor from using the "named transfer function" API and instead do everything in GPU shaders if no other KMS way exists, unless the KMS UAPI definition gives the exact mathematical formula of what that step will do.
For entertainment devices using Wayland as an internal detail, who cares. The system manufacturer makes things work any way they want, and you cannot run non-curated apps on those devices, so there are no compatibility issues.
Then there would be systems between those two extremes, which would likely use a mixture of those approaches, e.g. Kodi on bare hardware I imagine (no Wayland there, but Wayland is kind of unrelated except for the ideology).
Mind, I'm not talking only about EOTFs here, but thinking of the whole color management. EOTFs per se are simple. Tone-mapping, including gamut mapping and luminance mapping, are hard because there is no one correct formula.
However, if we are talking about *only* EOTFs, then an enumeration of the curves could be usable. But you were also talking about configuring dynamic ranges and doing luminance range mapping in the driver, which I think won't work with enums, but could perhaps be accounted for in M2 by userspace. Then again, if you have enums and not a LUT, you cannot implement custom non-linear per-channel mapping which might be useful for e.g. adapting SDR content for a HDR blending space. Or HDR content to a different HDR blending space.
See, a LUT can be more than just EOTF: it can apply a part of the tone-mapping too. An enum cannot.
I have a feeling that most tone-mapping curves would be non-linear with some sort of S-shape and clipping of black and white ends. But that is also a naive view, because tone-mapping is not a 1D-curve, it's a mapping from R^3 to R^3 in RGB since on a 600 nit monitor you cannot do 600 nit red for instance. On a 600 nit monitor you can do exactly one color with 600 nits: white.
M1 on AMD's driver is currently hidden in our driver and essentially always set to BT709 at the moment, though we should look at color_encoding to pick the right M1. Are the color_encoding and color_range properties enough for what you'd like to achieve for M1?
Probably yes, for standard format content, until a new standard comes up again and then you have to add another enum value in the UAPI. M1 is probably the best place where an enum fits in the pipeline though, and for M3 as well because non-RGB inputs to monitors are quite standardised, and at those steps I would assume there are no parts of tone-mapping done.
M3, currently the only matrix exposed by KMS (on drm_crtc) currently seems to be applied at M2 by AMD's driver. I need to dig into this a bit more. It would make sense to expose M2 and M3 as part of KMS (on drm_crtc and drm_plane).
Good, it's really important to make sure that all drivers agree on the pipeline layout that is defined(???) in the KMS UAPI specification.
I did a bunch of reading and thinking since your comments and created a v2 of the patchset. I don't think it's where we need it to be but I hope it's a step in the right direction. I will send it shortly.
I have to say I'm still very sceptical about this enum-based approach as you have noticed. :-)
Thanks, pq
On 2021-05-17 12:48 p.m., Sebastian Wick wrote:
On 2021-05-17 10:57, Pekka Paalanen wrote:
On Fri, 14 May 2021 17:05:11 -0400 Harry Wentland harry.wentland@amd.com wrote:
On 2021-04-27 10:50 a.m., Pekka Paalanen wrote:
On Mon, 26 Apr 2021 13:38:49 -0400 Harry Wentland harry.wentland@amd.com wrote:
...
## Mastering Luminances
Now we are able to use the PQ 2084 EOTF to define the luminance of pixels in absolute terms. Unfortunately we're again presented with physical limitations of the display technologies on the market
today.
Here are a few examples of luminance ranges of displays.
| Display | Luminance range in nits | | ------------------------ | ----------------------- | | Typical PC display | 0.3 - 200 | | Excellent LCD HDTV | 0.3 - 400 | | HDR LCD w/ local dimming | 0.05 - 1,500 |
Since no display can currently show the full 0.0005 to 10,000 nits luminance range the display will need to tonemap the HDR content,
i.e
to fit the content within a display's capabilities. To assist with tonemapping HDR content is usually accompanied with a metadata that describes (among other things) the minimum and maximum mastering luminance, i.e. the maximum and minimum luminance of the display
that
was used to master the HDR content.
The HDR metadata is currently defined on the drm_connector via the hdr_output_metadata blob property.
It might be useful to define per-plane hdr metadata, as different planes might have been mastered differently.
I don't think this would directly help with the dynamic range
blending
problem. You still need to establish the mapping between the optical values from two different EOTFs and dynamic ranges. Or can you know which optical values match the mastering display maximum and minimum luminances for not-PQ?
My understanding of this is probably best illustrated by this example:
Assume HDR was mastered on a display with a maximum white level of 500 nits and played back on a display that supports a max white level of 400 nits. If you know the mastering white level of 500 you know that this is the maximum value you need to compress down to 400 nits, allowing you to use the full extent of the 400 nits panel.
Right, but in the kernel, where do you get these nits values from?
hdr_output_metadata blob is infoframe data to the monitor. I think this should be independent of the metadata used for color transformations in the display pipeline before the monitor.
EDID may tell us the monitor HDR metadata, but again what is used in the color transformations should be independent, because EDIDs lie, lighting environments change, and users have different preferences.
What about black levels?
Do you want to do black level adjustment?
How exactly should the compression work?
Where do you map the mid-tones?
What if the end user wants something different?
I suspect that this is not about tone mapping at all. The use cases listed always have the display in PQ mode and just assume that no content exceeds the PQ limitations. Then you can simply bring all content to the color space with a matrix multiplication and then map the linear light content somewhere into the PQ range. Tone mapping is performed in the display only.
From a generic wayland compositor point of view this is uninteresting.
It a compositor's decision to provide or not the metadata property to the kernel. The metadata can be available from one or multiple clients or most likely not available at all.
Compositors may put a display in HDR10 ( when PQ 2084 INV EOTF and TM occurs in display ) or NATIVE mode and do not attach any metadata to the connector and do TM in compositor.
It is all about user preference or compositor design, or a combination of both options.
I completely agree with what you said below though. I would even argue that all generic KMS abstract pipeline elements must have a well defined place in the pipeline and follow an exact specified formula.
If you do not know the mastering luminance is 500 nits you would have to compress 10,000 nits down to 400 (assuming PQ), losing quite a bit of the full 400 nits available as you'll need room to map the 500 to 10,000 nits range which in reality is completely unused. You might end up with mapping 500 nits to 350 nits, instead of mapping it to 400.
The quality of the result depends on the compression (tone-mapping) algorithm. I believe no-one will ever do a simple linear compression of ranges.
Instead, you probably do something smooth in the black end, keep mid-tones roughly as they are, and the again do a smooth saturation to some "reasonable" level that goes well with the monitor in the current lighting environment without introducing coloring artifacts, and just clip the huge overshoot of the full PQ-range.
There are many big and small decisions to be made in how to map out-of-gamut or out-of-brightness values into the displayable range, and no algorithm fits all use cases. I believe this is why e.g. ICC has several different "render intents", some of which are so vaguely defined that they are practically undefined - just like what "a good picture" means. You have essentially three dimensions: luminance, hue, and saturation. Which one will you sacrifice, shift or emphasize and to what degree to fit the square content peg into the round display hole?
A naive example: Let's say content has 300 nits red. Your display can show max 400 nits white or max 180 nits red, and anything in between. What will you show?
The problem is, that if UAPI does not define exactly what happens, then taking advantage of these hardware capabilities means you have no idea what happens to your content. This may be fine for closed systems, where the software has been carefully built for the specific hardware revision and the use cases of the complete system have been pre-defined, so that it can assume what should and will happen. But should that be exposed as a generic UAPI, when generic userspace has no chance of knowing what it will do?
...
## Displays and Tonemapping
External displays are able to do their own tone and color mapping, based on the mastering luminance, color primaries, and white space defined in the HDR metadata.
Internal panels (which are currently few and far between) usually don't include the complex HW to do tone and color mapping on their own and will require the display driver to perform appropriate mapping.
FWIW, when designing Weston's color management, we are aiming for the latter "simple" panels foremost, because that gives us full
control
of all color conversions and tone mappings.
OTOH, if Weston has to present to a display which only accepts e.g. BT.2020/PQ signal, the display might always mangle the image in unexpected ways. Therefore I expect that by default Weston will do everything it can to try to make the display not apply anything magic image enhancement: trust that EDID description of the display
gamut and
dynamic range are correct, and use HDR metadata to tell the display that those values are exactly what we are using. And we use them.
IMO, a display doing its tone mapping magically is only useful
when you
want to be able to use "simple" playback devices that cannot adapt to the display they are driving. Magic tone mapping is also a way for hardware vendors to differentiate, which from the color management perspective is harmful as it makes it more difficult or impossible to predict the display behaviour or to keep it consistent.
So there are two opposing goals:
- Traditional color management wants absolute control of the display,
leaving nothing unpredictable and preferably also nothing
undefined.
Undefined behaviour can always be measured (profiled) which
makes it
predictable and known. The viewing environment is controlled and constant.
- Entertainment wants the most visually impressive image quality by
dynamically adapting to both displayed content and to the viewing environment conditions.
I feel much of the mess around HDR and CM is all the magic that everyone tries to do to differentiate themselves. I'm not a fan of that. occures
One thing to watch out for is that the definitions for some of these things (I'm thinking PWL LUTs, 3D LUT, SDR boots) might differ a fair bit between HW vendors and we'd want to find a way to hide these complexities to KMS clients.
How could we hide them?
Actually, what do you mean with "hide"?
...
**One ARGB8888 SDR Plane + One P010 HDR Plane:** A normal 8bpc desktop plane, with a P010 HDR video plane underlayed. The HDR plane shall be PQ BT.2020 formatted. The desktop plane shall specify an
SDR
boost value. The drm_connector's hdr_output_metadata shall be set.
This use case requires blending in KMS, so is the primary goal I suppose.
Correct, my understanding is that the key here is to ensure blending in KMS is done in same blending space (linear RGB) as blending using GPU shaders and SDR boost is applied in the same fashion.
Yes, but this is also something I think cannot be achieved by telling the kernel about color spaces and letting the driver come up with a color transformation. I believe this can only be achieved by telling the kernel the color transformation directly, and also describing to userspace what kind of color transformations are possible (1D LUT, 3D LUT, other; precision...).
...
## Summary of proposed interface changes
per drm_plane:
- new RGB color space definitions, mirroring the existing YUV color
space definitions
- new transfer function property
- new SDR maximum white level property
How will these new KMS properties interact with per-plane DEGAMMA,
CTM
and/or GAMMA properties?
Why go with your proposal instead of per-plane CTM and LUT?
I think the ideal KMS pipeline for me, assuming I cannot have 3D LUTs both per-plane and on CRTC, would be:
plane: FB -> M1 -> LUT1 -> M2 -> blending input
CRTC: blending output -> LUT2 -> M3 -> connector
FB: framebuffer M1: matrix transform, capable of converting e.g. YCbCr to RGB LUT1: 1D LUT for content EOTF, to produce light-linear RGB M2: matrix transform for color space transformation
LUT2: 1D LUT for applying monitor EOTF^-1 M3: matrix transform, e.g. if you need to push YCbCr on the connector
We also need to know where and how clipping happens.
I think this scheme would allow implementing everything you want, and it would not be tied to rigid enumerations, and it won't have any magical conversions done under the hood as you would need to do to convert from one enum space to another. It leaves the render
intent to
be defined by the userspace compositor, rather than building a fixed policy in the kernel.
Userspace would be setting transformation operators, not color
spaces,
to the kernel, allowing the blending space to be chosen by userspace. In Weston we aim to choose then blending color space to be the
same as
the output color space, except in optical (linear) encoding. The
output
color space can be non-standard, e.g. measured with a display
profiler
equipment.
I would expect gamut mapping, dynamic range mapping and tone
mapping to
be places where most experimentation and innovation happens, so implementing them in the kernel with just few or no controllable parameters at this time seems like it could become useless fast.
How important from a Wayland compositor perspective would it be to have fully configurable LUTs per plane, as opposed to providing KMS with a named transfer function, such as PQ or BT709, for LUT1 and LUT2?
For Wayland desktop compositors, absolutely essential in my opinion.
I would strongly advice against any desktop compositor from using the "named transfer function" API and instead do everything in GPU shaders if no other KMS way exists, unless the KMS UAPI definition gives the exact mathematical formula of what that step will do.
For entertainment devices using Wayland as an internal detail, who cares. The system manufacturer makes things work any way they want, and you cannot run non-curated apps on those devices, so there are no compatibility issues.
Then there would be systems between those two extremes, which would likely use a mixture of those approaches, e.g. Kodi on bare hardware I imagine (no Wayland there, but Wayland is kind of unrelated except for the ideology).
Mind, I'm not talking only about EOTFs here, but thinking of the whole color management. EOTFs per se are simple. Tone-mapping, including gamut mapping and luminance mapping, are hard because there is no one correct formula.
However, if we are talking about *only* EOTFs, then an enumeration of the curves could be usable. But you were also talking about configuring dynamic ranges and doing luminance range mapping in the driver, which I think won't work with enums, but could perhaps be accounted for in M2 by userspace. Then again, if you have enums and not a LUT, you cannot implement custom non-linear per-channel mapping which might be useful for e.g. adapting SDR content for a HDR blending space. Or HDR content to a different HDR blending space.
See, a LUT can be more than just EOTF: it can apply a part of the tone-mapping too. An enum cannot.
I have a feeling that most tone-mapping curves would be non-linear with some sort of S-shape and clipping of black and white ends. But that is also a naive view, because tone-mapping is not a 1D-curve, it's a mapping from R^3 to R^3 in RGB since on a 600 nit monitor you cannot do 600 nit red for instance. On a 600 nit monitor you can do exactly one color with 600 nits: white.
M1 on AMD's driver is currently hidden in our driver and essentially always set to BT709 at the moment, though we should look at color_encoding to pick the right M1. Are the color_encoding and color_range properties enough for what you'd like to achieve for M1?
Probably yes, for standard format content, until a new standard comes up again and then you have to add another enum value in the UAPI. M1 is probably the best place where an enum fits in the pipeline though, and for M3 as well because non-RGB inputs to monitors are quite standardised, and at those steps I would assume there are no parts of tone-mapping done.
M3, currently the only matrix exposed by KMS (on drm_crtc) currently seems to be applied at M2 by AMD's driver. I need to dig into this a bit more. It would make sense to expose M2 and M3 as part of KMS (on drm_crtc and drm_plane).
Good, it's really important to make sure that all drivers agree on the pipeline layout that is defined(???) in the KMS UAPI specification.
I did a bunch of reading and thinking since your comments and created a v2 of the patchset. I don't think it's where we need it to be but I hope it's a step in the right direction. I will send it shortly.
I have to say I'm still very sceptical about this enum-based approach as you have noticed. :-)
Thanks, pq
On Mon, 17 May 2021 15:39:03 -0400 Vitaly Prosyak vitaly.prosyak@amd.com wrote:
On 2021-05-17 12:48 p.m., Sebastian Wick wrote:
On 2021-05-17 10:57, Pekka Paalanen wrote:
On Fri, 14 May 2021 17:05:11 -0400 Harry Wentland harry.wentland@amd.com wrote:
On 2021-04-27 10:50 a.m., Pekka Paalanen wrote:
On Mon, 26 Apr 2021 13:38:49 -0400 Harry Wentland harry.wentland@amd.com wrote:
...
## Mastering Luminances
Now we are able to use the PQ 2084 EOTF to define the luminance of pixels in absolute terms. Unfortunately we're again presented with physical limitations of the display technologies on the market
today.
Here are a few examples of luminance ranges of displays.
| Display | Luminance range in nits | | ------------------------ | ----------------------- | | Typical PC display | 0.3 - 200 | | Excellent LCD HDTV | 0.3 - 400 | | HDR LCD w/ local dimming | 0.05 - 1,500 |
Since no display can currently show the full 0.0005 to 10,000 nits luminance range the display will need to tonemap the HDR content,
i.e
to fit the content within a display's capabilities. To assist with tonemapping HDR content is usually accompanied with a metadata that describes (among other things) the minimum and maximum mastering luminance, i.e. the maximum and minimum luminance of the display
that
was used to master the HDR content.
The HDR metadata is currently defined on the drm_connector via the hdr_output_metadata blob property.
It might be useful to define per-plane hdr metadata, as different planes might have been mastered differently.
I don't think this would directly help with the dynamic range
blending
problem. You still need to establish the mapping between the optical values from two different EOTFs and dynamic ranges. Or can you know which optical values match the mastering display maximum and minimum luminances for not-PQ?
My understanding of this is probably best illustrated by this example:
Assume HDR was mastered on a display with a maximum white level of 500 nits and played back on a display that supports a max white level of 400 nits. If you know the mastering white level of 500 you know that this is the maximum value you need to compress down to 400 nits, allowing you to use the full extent of the 400 nits panel.
Right, but in the kernel, where do you get these nits values from?
hdr_output_metadata blob is infoframe data to the monitor. I think this should be independent of the metadata used for color transformations in the display pipeline before the monitor.
EDID may tell us the monitor HDR metadata, but again what is used in the color transformations should be independent, because EDIDs lie, lighting environments change, and users have different preferences.
What about black levels?
Do you want to do black level adjustment?
How exactly should the compression work?
Where do you map the mid-tones?
What if the end user wants something different?
I suspect that this is not about tone mapping at all. The use cases listed always have the display in PQ mode and just assume that no content exceeds the PQ limitations. Then you can simply bring all content to the color space with a matrix multiplication and then map the linear light content somewhere into the PQ range. Tone mapping is performed in the display only.
The use cases do use the word "desktop" though. Harry, could you expand on this, are you seeking a design that is good for generic desktop compositors too, or one that is more tailored to "embedded" video player systems taking the most advantage of (potentially fixed-function) hardware?
What matrix would one choose? Which render intent would it correspond to?
If you need to adapt different dynamic ranges into the blending dynamic range, would a simple linear transformation really be enough?
From a generic wayland compositor point of view this is uninteresting.
It a compositor's decision to provide or not the metadata property to the kernel. The metadata can be available from one or multiple clients or most likely not available at all.
Compositors may put a display in HDR10 ( when PQ 2084 INV EOTF and TM occurs in display ) or NATIVE mode and do not attach any metadata to the connector and do TM in compositor.
It is all about user preference or compositor design, or a combination of both options.
Indeed. The thing here is that you cannot just add KMS UAPI, you also need to have the FOSS userspace to go with it. So you need to have your audience defined, userspace patches written and reviewed and agreed to be a good idea. I'm afraid this particular UAPI design would be difficult to justify with Weston. Maybe Kodi is a better audience?
But then again, one also needs to consider whether it is enough for a new UAPI to satisfy only part of the possible audience and then need yet another new UAPI to satisfy the rest. Adding new UAPI requires defining the interactions with all existing UAPI as well.
Maybe we do need several different UAPIs for the "same" things if the hardware designs are too different to cater with just one.
Thanks, pq
On 2021-05-18 3:56 a.m., Pekka Paalanen wrote:
On Mon, 17 May 2021 15:39:03 -0400 Vitaly Prosyak vitaly.prosyak@amd.com wrote:
On 2021-05-17 12:48 p.m., Sebastian Wick wrote:
On 2021-05-17 10:57, Pekka Paalanen wrote:
On Fri, 14 May 2021 17:05:11 -0400 Harry Wentland harry.wentland@amd.com wrote:
On 2021-04-27 10:50 a.m., Pekka Paalanen wrote:
On Mon, 26 Apr 2021 13:38:49 -0400 Harry Wentland harry.wentland@amd.com wrote:
...
> ## Mastering Luminances > > Now we are able to use the PQ 2084 EOTF to define the luminance of > pixels in absolute terms. Unfortunately we're again presented with > physical limitations of the display technologies on the market
today.
> Here are a few examples of luminance ranges of displays. > > | Display | Luminance range in nits | > | ------------------------ | ----------------------- | > | Typical PC display | 0.3 - 200 | > | Excellent LCD HDTV | 0.3 - 400 | > | HDR LCD w/ local dimming | 0.05 - 1,500 | > > Since no display can currently show the full 0.0005 to 10,000 nits > luminance range the display will need to tonemap the HDR content,
i.e
> to fit the content within a display's capabilities. To assist with > tonemapping HDR content is usually accompanied with a metadata that > describes (among other things) the minimum and maximum mastering > luminance, i.e. the maximum and minimum luminance of the display
that
> was used to master the HDR content. > > The HDR metadata is currently defined on the drm_connector via the > hdr_output_metadata blob property. > > It might be useful to define per-plane hdr metadata, as different > planes might have been mastered differently.
I don't think this would directly help with the dynamic range
blending
problem. You still need to establish the mapping between the optical values from two different EOTFs and dynamic ranges. Or can you know which optical values match the mastering display maximum and minimum luminances for not-PQ?
My understanding of this is probably best illustrated by this example:
Assume HDR was mastered on a display with a maximum white level of 500 nits and played back on a display that supports a max white level of 400 nits. If you know the mastering white level of 500 you know that this is the maximum value you need to compress down to 400 nits, allowing you to use the full extent of the 400 nits panel.
Right, but in the kernel, where do you get these nits values from?
hdr_output_metadata blob is infoframe data to the monitor. I think this should be independent of the metadata used for color transformations in the display pipeline before the monitor.
EDID may tell us the monitor HDR metadata, but again what is used in the color transformations should be independent, because EDIDs lie, lighting environments change, and users have different preferences.
What about black levels?
Do you want to do black level adjustment?
How exactly should the compression work?
Where do you map the mid-tones?
What if the end user wants something different?
I suspect that this is not about tone mapping at all. The use cases listed always have the display in PQ mode and just assume that no content exceeds the PQ limitations. Then you can simply bring all content to the color space with a matrix multiplication and then map the linear light content somewhere into the PQ range. Tone mapping is performed in the display only.
The use cases do use the word "desktop" though. Harry, could you expand on this, are you seeking a design that is good for generic desktop compositors too, or one that is more tailored to "embedded" video player systems taking the most advantage of (potentially fixed-function) hardware?
The goal is to enable this on a generic desktop, such as generic Wayland implementations or ChromeOS. We're not looking for a custom solution for some embedded systems, though the solution we end up with should obviously not prevent an implementation on embedded video players.
What matrix would one choose? Which render intent would it correspond to?
If you need to adapt different dynamic ranges into the blending dynamic range, would a simple linear transformation really be enough?
From a generic wayland compositor point of view this is uninteresting.
It a compositor's decision to provide or not the metadata property to the kernel. The metadata can be available from one or multiple clients or most likely not available at all.
Compositors may put a display in HDR10 ( when PQ 2084 INV EOTF and TM occurs in display ) or NATIVE mode and do not attach any metadata to the connector and do TM in compositor.
It is all about user preference or compositor design, or a combination of both options.
Indeed. The thing here is that you cannot just add KMS UAPI, you also need to have the FOSS userspace to go with it. So you need to have your audience defined, userspace patches written and reviewed and agreed to be a good idea. I'm afraid this particular UAPI design would be difficult to justify with Weston. Maybe Kodi is a better audience?
I'm not sure designing a UAPI for Kodi that's not going to work for Wayland-compositors is the right thing. From a KMS driver maintainer standpoint I don't want an API for each userspace.
The idea here is to do design and discussion in public so we can eventually arrive at a UAPI for HDR and CM that works for Wayland and by extension for every other userspace.
But then again, one also needs to consider whether it is enough for a new UAPI to satisfy only part of the possible audience and then need yet another new UAPI to satisfy the rest. Adding new UAPI requires defining the interactions with all existing UAPI as well.
Maybe we do need several different UAPIs for the "same" things if the hardware designs are too different to cater with just one.
I feel we should have a section in the RFC that sketches how different HW deals with this currently. It would be good if we can arrive at a UAPI that captures at least the common functionality of various HW.
Harry
Thanks, pq
On 2021-05-18 16:19, Harry Wentland wrote:
On 2021-05-18 3:56 a.m., Pekka Paalanen wrote:
On Mon, 17 May 2021 15:39:03 -0400 Vitaly Prosyak vitaly.prosyak@amd.com wrote:
On 2021-05-17 12:48 p.m., Sebastian Wick wrote:
On 2021-05-17 10:57, Pekka Paalanen wrote:
On Fri, 14 May 2021 17:05:11 -0400 Harry Wentland harry.wentland@amd.com wrote:
On 2021-04-27 10:50 a.m., Pekka Paalanen wrote: > On Mon, 26 Apr 2021 13:38:49 -0400 > Harry Wentland harry.wentland@amd.com wrote:
...
>> ## Mastering Luminances >> >> Now we are able to use the PQ 2084 EOTF to define the luminance >> of >> pixels in absolute terms. Unfortunately we're again presented >> with >> physical limitations of the display technologies on the market today. >> Here are a few examples of luminance ranges of displays. >> >> | Display | Luminance range in nits | >> | ------------------------ | ----------------------- | >> | Typical PC display | 0.3 - 200 | >> | Excellent LCD HDTV | 0.3 - 400 | >> | HDR LCD w/ local dimming | 0.05 - 1,500 | >> >> Since no display can currently show the full 0.0005 to 10,000 >> nits >> luminance range the display will need to tonemap the HDR >> content, i.e >> to fit the content within a display's capabilities. To assist >> with >> tonemapping HDR content is usually accompanied with a metadata >> that >> describes (among other things) the minimum and maximum mastering >> luminance, i.e. the maximum and minimum luminance of the display that >> was used to master the HDR content. >> >> The HDR metadata is currently defined on the drm_connector via >> the >> hdr_output_metadata blob property. >> >> It might be useful to define per-plane hdr metadata, as >> different >> planes might have been mastered differently. > > I don't think this would directly help with the dynamic range blending > problem. You still need to establish the mapping between the > optical > values from two different EOTFs and dynamic ranges. Or can you > know > which optical values match the mastering display maximum and > minimum > luminances for not-PQ? >
My understanding of this is probably best illustrated by this example:
Assume HDR was mastered on a display with a maximum white level of 500 nits and played back on a display that supports a max white level of 400 nits. If you know the mastering white level of 500 you know that this is the maximum value you need to compress down to 400 nits, allowing you to use the full extent of the 400 nits panel.
Right, but in the kernel, where do you get these nits values from?
hdr_output_metadata blob is infoframe data to the monitor. I think this should be independent of the metadata used for color transformations in the display pipeline before the monitor.
EDID may tell us the monitor HDR metadata, but again what is used in the color transformations should be independent, because EDIDs lie, lighting environments change, and users have different preferences.
What about black levels?
Do you want to do black level adjustment?
How exactly should the compression work?
Where do you map the mid-tones?
What if the end user wants something different?
I suspect that this is not about tone mapping at all. The use cases listed always have the display in PQ mode and just assume that no content exceeds the PQ limitations. Then you can simply bring all content to the color space with a matrix multiplication and then map the linear light content somewhere into the PQ range. Tone mapping is performed in the display only.
The use cases do use the word "desktop" though. Harry, could you expand on this, are you seeking a design that is good for generic desktop compositors too, or one that is more tailored to "embedded" video player systems taking the most advantage of (potentially fixed-function) hardware?
The goal is to enable this on a generic desktop, such as generic Wayland implementations or ChromeOS. We're not looking for a custom solution for some embedded systems, though the solution we end up with should obviously not prevent an implementation on embedded video players.
What matrix would one choose? Which render intent would it correspond to?
If you need to adapt different dynamic ranges into the blending dynamic range, would a simple linear transformation really be enough?
From a generic wayland compositor point of view this is uninteresting.
It a compositor's decision to provide or not the metadata property to the kernel. The metadata can be available from one or multiple clients or most likely not available at all.
Compositors may put a display in HDR10 ( when PQ 2084 INV EOTF and TM occurs in display ) or NATIVE mode and do not attach any metadata to the connector and do TM in compositor.
It is all about user preference or compositor design, or a combination of both options.
Indeed. The thing here is that you cannot just add KMS UAPI, you also need to have the FOSS userspace to go with it. So you need to have your audience defined, userspace patches written and reviewed and agreed to be a good idea. I'm afraid this particular UAPI design would be difficult to justify with Weston. Maybe Kodi is a better audience?
I'm not sure designing a UAPI for Kodi that's not going to work for Wayland-compositors is the right thing. From a KMS driver maintainer standpoint I don't want an API for each userspace.
The idea here is to do design and discussion in public so we can eventually arrive at a UAPI for HDR and CM that works for Wayland and by extension for every other userspace.
Eventually we want to be able to drive displays in PQ mode in weston (I think?) where the TF property could be used in some cases. So that property seems good to me. The SDR boost property then might also be useful but it depends on the details of the formula/mechanism that you can come up with.
But I want to stress again that we're going to drive all display in their native color space and dynamic range if possible where none of those properties discussed are useful and without some kind of 3D LUT in the plane's pixel pipeline we won't be able to make use of them at all.
The Kodi folks can probably give you better feedback and an actual implementation in reasonable time because they want to drive the display in PQ mode.
But then again, one also needs to consider whether it is enough for a new UAPI to satisfy only part of the possible audience and then need yet another new UAPI to satisfy the rest. Adding new UAPI requires defining the interactions with all existing UAPI as well.
Maybe we do need several different UAPIs for the "same" things if the hardware designs are too different to cater with just one.
I feel we should have a section in the RFC that sketches how different HW deals with this currently. It would be good if we can arrive at a UAPI that captures at least the common functionality of various HW.
Harry
Thanks, pq
On Tue, 18 May 2021 10:19:25 -0400 Harry Wentland harry.wentland@amd.com wrote:
On 2021-05-18 3:56 a.m., Pekka Paalanen wrote:
On Mon, 17 May 2021 15:39:03 -0400 Vitaly Prosyak vitaly.prosyak@amd.com wrote:
On 2021-05-17 12:48 p.m., Sebastian Wick wrote:
...
I suspect that this is not about tone mapping at all. The use cases listed always have the display in PQ mode and just assume that no content exceeds the PQ limitations. Then you can simply bring all content to the color space with a matrix multiplication and then map the linear light content somewhere into the PQ range. Tone mapping is performed in the display only.
The use cases do use the word "desktop" though. Harry, could you expand on this, are you seeking a design that is good for generic desktop compositors too, or one that is more tailored to "embedded" video player systems taking the most advantage of (potentially fixed-function) hardware?
The goal is to enable this on a generic desktop, such as generic Wayland implementations or ChromeOS. We're not looking for a custom solution for some embedded systems, though the solution we end up with should obviously not prevent an implementation on embedded video players.
(There is a TL;DR: at the end.)
Echoing a little bit what Sebastian already said, I believe there are two sides to this again: - color management in the traditional sense - modern standardised display technology
It was perhaps too harsh to say that generic Wayland compositors cannot use enum-based color-related UAPI. Sometimes they could, sometimes it won't be good enough.
Traditional color management assumes that no two monitors are the same, even if they are the same make, model, and manufacturing batch, and are driven exactly the same way. Hence, all monitors may require calibration (adjusting monitor knobs), and/or they may require profiling (measuring the light emission with a special hardware device designed for that). Also the viewing environment has an effect.
For profiling to be at all meaningful, calibration must be fixed. This means that there must be no dynamic on-the-fly adaptation done in the monitor, in the display hardware, or in the kernel. That is a tall order that I guess is going to be less and less achievable, especially with HDR monitors.
The other side is where the end user trusts the standards, and trusts that the drivers and the hardware do what they are specified to do. This is where you can trust that the monitor does the tone-mapping magic right.
Weston needs to support both approaches, because we want to prove our new approach to traditional color management, but we also want to support HDR, and if possible, do both at the same time. Doing both at the same time is what we think foremost, because it's also the hardest thing to achieve. If that can be done, then everything else works out too.
However, this should not exclude the possibility to trust standards and monitor magic, when the end user wants it.
It's also possible that a monitor simply doesn't support a mode that would enable fully color managed HDR, so Weston will need to be able to drive monitors with e.g. BT.2020/PQ data eventually. It's just not the first goal we have.
This debate is a little bit ironic. The Wayland approach to traditional color management is that end users should trust the display server to do the right thing, where before people only trusted the individual apps using a specific CMS implementation. The display server was the untrusted one that should just get out of the way and not touch anything. Now I'm arguing that I don't want to trust monitor magic, who knows what atrocities it does to my picture! But take the next logical step, and one would be arguing that end users should trust also monitors to do the right thing. :-)
The above has two catches:
- Do you actually trust hardware manufacturers and marketers and EDID? Monitors have secret sauce you can't inspect nor change.
- You feed a single video stream to a monitor, in a single format, encoding and color space. The display server OTOH gets an arbitrary number of input video streams in arbitrary formats, encodings, and color spaces, and it needs to composite them into one.
Composition is hard. It's not enough to know what kind of signals you take in and what kind of signal you must output. You also need to know what the end user wants from the result: the render intent.
Even if we trust the monitor magic to do the right thing in interpreting and displaying our output signal, we still need to know what the end user wants from the composition, and we need to control the composition formula to achieve that.
TL;DR:
I would summarise my comments so far into these:
- Telling the kernel the color spaces and letting it come up with whatever color transformation formula from those is not enough, because it puts the render intent policy decision in the kernel.
- Telling the kernel what color transformations need to be done is good, if it is clearly defined.
- Using an enum-based UAPI to tell the kernel what color transformations needs to be done (e.g. which EOTF or EOTF^-1 to apply at a step in the abstract pipeline) is very likely ok for many Wayland compositors in most cases, but may not be sufficient for all use cases. Of course, one is always bound by what hardware can do, so not a big deal.
- You may need to define mutually exclusive KMS properties (referring to my email in another branch of this email tree).
- I'm not sure I (we?) can meaningfully review things like "SDR boost" property until we know ourselves how to composite different types of content together. Maybe someone else could.
Does this help or raise thoughts?
The work on Weston CM&HDR right now is aiming to get it up to a point where we can start nicely testing different compositing approaches and methods and parameters, and I expect that will also feed back into the Wayland CM&HDR protocol design as well.
Thanks, pq
On Wed, 19 May 2021 11:53:37 +0300 Pekka Paalanen ppaalanen@gmail.com wrote:
...
TL;DR:
I would summarise my comments so far into these:
Telling the kernel the color spaces and letting it come up with whatever color transformation formula from those is not enough, because it puts the render intent policy decision in the kernel.
Telling the kernel what color transformations need to be done is good, if it is clearly defined.
Using an enum-based UAPI to tell the kernel what color transformations needs to be done (e.g. which EOTF or EOTF^-1 to apply at a step in the abstract pipeline) is very likely ok for many Wayland compositors in most cases, but may not be sufficient for all use cases. Of course, one is always bound by what hardware can do, so not a big deal.
You may need to define mutually exclusive KMS properties (referring to my email in another branch of this email tree).
I'm not sure I (we?) can meaningfully review things like "SDR boost" property until we know ourselves how to composite different types of content together. Maybe someone else could.
Does this help or raise thoughts?
The work on Weston CM&HDR right now is aiming to get it up to a point where we can start nicely testing different compositing approaches and methods and parameters, and I expect that will also feed back into the Wayland CM&HDR protocol design as well.
I have forgot to mention one important thing:
Generic Wayland compositors will be using KMS planes opportunistically. The compositor will be switching between GL and KMS compositing on-demand, refresh by refresh. This means that both GL and KMS compositing must produce identical results, or users will be seeing "color flicks" on switch.
This is a practical reason why we really want to know in full detail how the KMS pipeline processes pixels.
Thanks, pq
dri-devel@lists.freedesktop.org
-
Daniel Vetter -
Harry Wentland -
Pekka Paalanen -
Sebastian Wick -
Shashank Sharma -
Ville Syrjälä -
Vitaly Prosyak