[RFC] Plane color pipeline KMS uAPI

Tue May 9 08:04:25 UTC 2023

On Thu, 04 May 2023 15:22:59 +0000
Simon Ser <contact at emersion.fr> wrote:

> Hi all,
> 
> The goal of this RFC is to expose a generic KMS uAPI to configure the color
> pipeline before blending, ie. after a pixel is tapped from a plane's
> framebuffer and before it's blended with other planes. With this new uAPI we
> aim to reduce the battery life impact of color management and HDR on mobile
> devices, to improve performance and to decrease latency by skipping
> composition on the 3D engine. This proposal is the result of discussions at
> the Red Hat HDR hackfest [1] which took place a few days ago. Engineers
> familiar with the AMD, Intel and NVIDIA hardware have participated in the
> discussion.
> 
> This proposal takes a prescriptive approach instead of a descriptive approach.
> Drivers describe the available hardware blocks in terms of low-level
> mathematical operations, then user-space configures each block. We decided
> against a descriptive approach where user-space would provide a high-level
> description of the colorspace and other parameters: we want to give more
> control and flexibility to user-space, e.g. to be able to replicate exactly the
> color pipeline with shaders and switch between shaders and KMS pipelines
> seamlessly, and to avoid forcing user-space into a particular color management
> policy.
> 
> We've decided against mirroring the existing CRTC properties
> DEGAMMA_LUT/CTM/GAMMA_LUT onto KMS planes. Indeed, the color management
> pipeline can significantly differ between vendors and this approach cannot
> accurately abstract all hardware. In particular, the availability, ordering and
> capabilities of hardware blocks is different on each display engine. So, we've
> decided to go for a highly detailed hardware capability discovery.
> 
> This new uAPI should not be in conflict with existing standard KMS properties,
> since there are none which control the pre-blending color pipeline at the
> moment. It does conflict with any vendor-specific properties like
> NV_INPUT_COLORSPACE or the patches on the mailing list adding AMD-specific
> properties. Drivers will need to either reject atomic commits configuring both
> uAPIs, or alternatively we could add a DRM client cap which hides the vendor
> properties and shows the new generic properties when enabled.

Hi,

I have some further ideas which do conflict with some existing KMS
properties. This dives into the color encoding specific side of the
UAPI.

The main idea is to make the color pipeline not specific to RGB. We
might actually be having YCbCr, XYZ, ICtCp and whatnot instead, at
least in the middle of a pipeline. The aim is to avoid the confusion
from statements like "my red channel is actually luma and not red". So
it's purely syntactic. ISTR some people being against saying "R is just
a channel name, it's not necessarily a red component."

Therefore I propose to address the color channels with indices instead:

	ch 0, ch 1, ch 2, ch 3

Then we define the mapping between pixel and wire formats and the
indices:

	R = ch 0
	G = ch 1
	B = ch 2
	A = ch 3

	Y = ch 0
	U = ch 1
	V = ch 2

If necessary, the following can also be defined:

	Z = ch 1
	X = ch 2
	L = ch 0
	M = ch 1
	S = ch 2

The Y from YUV and Y from XYZ share the designation for the name's sake
although they are not the same quantity. If YUV is not a well-defined
designation wrt. YCbCr, ICtCp and everything else in the same category,
we can assign Cb, Cr, I, Ct, Cp etc. instead. That might be more clear
anyway even if there is a popular convention.

We can also choose differently, to e.g. match the H.273 mapping where
channels are assigned such that Y=G, Cr=R, Cb=B. H.273 gives mappings
between almost all of these, so if using those make more sense, then
let's use those. In the end it shouldn't matter too much, since one
does not arbitrarily mix channels from different formats. Special care
needs to be taken when defining COLOROP elements that do not handle all
channels interchangeably. (E.g. a curve set element is mostly
channel-agnostic when it applies the same curve to channels 0-2, but ch
3 is pass-through.)

Then, we define COLOROP elements in terms of channel indices. This
removes any implied connection to any specific color coding. Elements
that just do not make sense for arbitrary channel ordering, e.g.
special-case elements or enumerated matrix elements with a specific
purpose, will document the intended usage and the expected channel
mapping.

The main reason to do all this is to ultimately allow e.g. limited
range YCbCr scanout with a fully pass-through pipeline with no implied
conversion to or from RGB.

This is where some existing KMS properties will conflict: those that
affect how current implicit YUV-RGB conversions are done. These
properties shall be replaced with COLOROP elements in pipelines, so
that they can be controlled explicitly and we can know where they
reside wrt. e.g. sampling operations. Chroma channel reconstruction
from sub-sampled chroma planes could potentially be explicitly
represented, and so it could also be controlled (chroma siting).

"Nominal" or "normalized" color value encoding at the input and output
of each COLOROP element needs to be defined as well. Some elements,
like matrices, can theoretically handle arbitrary values, but some
elements like LUTs are inherently limited in their input values.
Regardless of how a LUT is defined, it almost always assumes an input
range [0.0, 1.0] of nominal color value encoding. This is particularly
important for cases where out-of-unit-range values are expected:
- scRGB
- maybe some HDR encodings
- YCbCr and anything else with chroma which is normally [-0.5, 0.5]
- xvYCC

Most likely we need to find some way to map everything to the nominal
range [0.0, 1.0] if at all possible. We also need to know how elements
handle input values outside of that range.

The KMS property to control the scaling filter would be replaced
by the scaling COLOROP element with changeable filter property.

All this applies to pre-blending pipelines already. For the
post-blending pipeline we have the added complication of drivers
automatically choosing the wire format, but maybe that can be
hand-waved with a special-purpose COLOROP element that does whatever
the driver chooses, until it can be fully controlled from userspace.

Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20230509/3e6c21f3/attachment-0001.sig>