The state of Quantization Range handling

[Pekka Paalanen]

On Tue, 15 Nov 2022 00:11:56 +0100
Sebastian Wick <sebastian.wick at redhat.com> wrote:

> There are still regular bug reports about monitors (sinks) and sources
> disagreeing about the quantization range of the pixel data. In
> particular sources sending full range data when the sink expects
> limited range. From a user space perspective, this is all hidden in
> the kernel. We send full range data to the kernel and then hope it
> does the right thing but as the bug reports show: some combinations of
> displays and drivers result in problems.
> 
> In general the whole handling of the quantization range on linux is
> not defined or documented at all. User space sends full range data
> because that's what seems to work most of the time but technically
> this is all undefined and user space can not fix those issues. Some
> compositors have resorted to giving users the option to choose the
> quantization range but this really should only be necessary for
> straight up broken hardware.
> 
> Quantization Range can be explicitly controlled by AVI InfoFrame or
> HDMI General Control Packets. This is the ideal case and when the
> source uses them there is not a lot that can go wrong. Not all
> displays support those explicit controls in which case the chosen
> video format (IT, CE, SD; details in CTA-861-H 5.1) influences which
> quantization range the sink expects.
> 
> This means that we have to expect that sometimes we have to send
> limited and sometimes full range content. The big question however
> that is not answered in the docs: who is responsible for making sure
> the data is in the correct range? Is it the kernel or user space?
> 
> If it's the kernel: does user space supply full range or limited range
> content? Each of those has a disadvantage. If we send full range
> content and the driver scales it down to limited range, we can't use
> the out-of-range bits to transfer information. If we send limited
> range content and the driver scales it up we lose information.
> 
> Either way, this must be documented. My suggestion is to say that the
> kernel always expects full range data as input and is responsible for
> scaling it to limited range data if the sink expects limited range
> data.

Hi Sebastian,

you are proposing the that driver/hardware will do either no range
conversion, or full-to-limited range conversion. Limited-to-full range
conversion would never be supported.

I still wonder if limited-to-full range conversion could be useful with
video content.

> Another problem is that some displays do not behave correctly. It must
> be possible to override the kernel when the user detects such a
> situation. This override then controls if the driver converts the full
> range data coming from the client or not (Default, Force Limited,
> Force Full). It does not try to control what range the sink expects.
> Let's call this the Quantization Range Override property which should
> be implemented by all drivers.

In other words, a CRTC "quantization range conversion" property with
values:
- auto, with the assumption that color pipeline always produces full-range
- identity
- full-to-limited
(- limited-to-full)

If this property was truly independent of the metadata being sent to
the sink, and of the framebuffer format, it would allow us to do four
ways: both full/limited framebuffer on both full/limited sink. It would
allow us to send sub-blacks and super-whites as well.

More precisely, framebuffers would always have *undefined* quantization
range. The configuration of the color pipeline then determines how that
data is manipulated into a video signal.

So I am advocating the same design as with color spaces: do not tell
KMS what your colorspaces are. Instead tell KMS what operations it
needs to do with the pixel data, and what metadata to send to the sink.

> All drivers should make sure their behavior is correct:
> 
> * check that drivers choose the correct default quantization range for
> the selected mode

Mode implying a quantization range is awkward, but maybe the kernel
established modes should just have a flag for it. Then userspace would
know. Unless the video mode system is extended to communicate
IT/CE/SD/VIC and whatnot to userspace, making the modes better defined.
Then userspace would know too.

> * whenever explicit control is available, use it and set the
> quantization range to full
> * make sure that the hardware converts from full range to limited
> range whenever the sink expects limited range
> * implement the Quantization Range Override property
> 
> I'm volunteering for the documentation, UAPI and maybe even the drm
> core parts if there is willingness to tackle the issue.

Is it a good idea to put even more automation/magic into configuring
the color pipeline and metadata for a sink, making them even more
intertwined?

I would prefer the opposite direction, making thing more explicit and
orthogonal.


Thanks,
pq

> Appendix A: Broadcast RGB property
> 
> A few drivers already implement the Broadcast RGB property to control
> the quantization range. However, it is pointless: It can be set to
> Auto, Full and Limited when the sink supports explicitly setting the
> quantization range. The driver expects full range content and converts
> it to limited range content when necessary. Selecting limited range
> never makes any sense: the out-of-range bits can't be used because the
> input is full range. Selecting Default never makes sense: relying on
> the default quantization range is risky because sinks often get it
> wrong and as we established there is no reason to select limited range
> if not necessary. The limited and full options also are not suitable
> as an override because the property is not available if the sink does
> not support explicitly setting the quantization range.
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20221116/4d2490f7/attachment.sig>