HDR support in Wayland/Weston

Tue Jan 29 11:08:28 UTC 2019

On Tue, 29 Jan 2019 16:02:16 +1100
Graeme Gill <graeme2 at argyllcms.com> wrote:

> Pekka Paalanen wrote:
> 

> >>> Yes, a compositor must implement all that, but this is now slipping to
> >>> the topic of calibration, which is very much off-scope for today. We
> >>> only want to consider how applications produce and provide content with
> >>> specific color properties for now.  
> 
> >> It's a necessary part of the picture. There's not much point in
> >> moving ahead with Color Management support if there is no
> >> easy means of creating display profiles to populate it with. So in
> >> terms of practical implementation I see them going hand in hand.  
> > 
> > They may go hand-in-hand in practise, but protocol-wise I still see
> > them as two completely orthogonal features, and we need to split the
> > work into manageable chunks anyway.  
> 
> I don't have any basis to voice an opinion as to which particular protocol
> these different aspects would best fall into (wayland core or one of the
> supplementary xdg protocols ? - I'm not clear on what the purpose of these
> divisions is), I just know that they are intimately related, and development
> of one without the other will risk mis-steps. I don't really understand
> why you think they are orthogonal.

Hi Graeme,

color management related extensions would be neither Wayland core (in
wayland.xml file) nor under the XDG umbrella (XDG is specific to
desktops while color is not specific to desktops).

Color management related extensions, just like presentation-time and
viewporter, would live in wayland-protocols repository where they are
easily shared by all compositor and client software that wants to use
them. The Wayland upstream cannot force anyone to implement any
extension. The upstream can offer design help, reviews, discussion
forum and a repository to host extensions, which should make extensions
popular among client software and suitable for more than one
compositor. The pressure for someone to implement an extension only
ever comes from the community: people wanting new things to work.

Color management related extensions would be several different
extensions, each catering to its own scope. The one we are talking
about here is for clients to be able to reasonably provide color
managed content and enable the correct displaying of it under automatic
adaptation to outputs. Other color management related extensions could
allow e.g. measuring the color profiles of monitors, to be discussed at
another time.

Being able to provide color managed content implies knowing the color
properties of the used hardware and images. It does not imply being
able to measure the color properties: you don't re-measure the color
properties continuously while using an application. Or if you actually
do, we can cater for that use case when designing a measurement
extension.

> > Surely the color characterisation of a monitor is not specific to a
> > window system?  
> 
> Very much so, since in practice this process relies on running an application
> that makes use of the window system for its operation.

I don't understand. How is a color profile, or say, an .icc file or
whatever you use to store color profiles, dependent on the window
system? Do you mean you cannot use the same profile data files on
Windows and Linux X11?

Or do you mean that the window system implementation adds its own
unknown effect to the pixel values? If that is so, then the measuring
environment on those systems is fundamentally broken to begin with.

> > While we still don't have a good solution to
> > measurements in Wayland, people can measure their monitors on Xorg, or
> > even Windows, I hope.  
> 
> It's unsafe to switch pixel pipelines in the process of characterization,
> since it risks measuring transformations that happen/don't happen in
> one context, and not in the other. It's also not something that
> end users will put up with - if a system claims to support color management,
> then it includes the color management tools necessary to setup,
> maintain and verify its operation on a day to day basis.

That seems like an unrealistic assumption, it kind of precludes the
whole idea of interoperability. If your assumption is held, then you
would need to make separate measurements for e.g. every single pixel
format × color space an application is going to provide content to the
compositor in, multiplied by all the different ways a compositor can
make use of the graphics card hardware to produce the video signal on
that single system, and you get to redo this every time you update any
software component.

I would rather aim towards a design where you measure your monitor (and
not the window system), and the compositor will take care of keeping
that measured profile valid regardless of how the compositor chooses to
use the hardware. I would put responsibility to the compositor and
hardware drivers to function correctly, instead of requiring every
application needing to verify the whole system every time it starts up.
This will also leave compositor implementations more room to make the
most out of the hardware at hand, rather than forcing them into a
single specific pixel pipeline than can never change.

> From a development point of view it's working in a an unnecessary
> difficult manner, since there will be nothing to exercise the full
> range of color management capabilities, or measurably verify
> the implementation against.

Things will come piece by piece. Things will get tested and verified
piece by piece. Bugs will be introduced and fixed one by one. Trying to
design everything at once and implement everything at once makes no
difference: people still work piece by piece.

I think it's better to acknowledge that people work piece by piece and
things may have to change in the future once more pieces come together.
Wayland-protocols repository rules also have this workflow defined:
extensions start as unstable, can be revised and changed radically,
until they are agreed to be stable. We can just agree to not call color
management related extensions stable until we have also measuring
covered.

> > Or even better: using a measuring tool running
> > directly on DRM KMS, which ensures the tool gets absolute control over
> > the display hardware.  
> 
> Definitely the last thing you want, since that leaves a gaping
> hole for the hardware to be setup differently between profiling
> and use of the profile, thereby making the profile invalid.
> The safe approach is to maintain as much similarity as possible
> between system setup while profiling and when the profile
> is used. Ensuring that a profile remains valid from
> profiling to use is a key consideration in color management,
> and without this, it all falls to pieces.

100% similarity is not realistic in any case.

I totally agree on the usability of a measured color profile, which is
why I would like it to describe the output device (a monitor in this
case) and nothing else.

> Completely made up example: Say the display is actually
> a TV running over HDMI, and (for whatever reason) the
> compositor sets up the link in YCbCr mode rather than RGB,
> and as luck would have it, the video card encodes the framebuffer
> RGB using REC 709 1150/60/2:1 standard, while the TV manufacturer
> didn't bother supporting the various YCbCr encodings and hard
> coded it to decode Rec601 YCbCr. So there is a hidden color
> change going on when Wayland is running as well as some
> minor quantization errors being introduced. But if the color
> profiling application was to configure the hardware, it
> may instead setup the HDMI in RGB mode (being the one
> with the highest fidelity), and the color response won't
> be the same as Wayland, so the profile made this way will
> be invalid. Ensuring the graphics card and every other
> element in the rendering pipeline is setup the same way,
> by the same software, this hard to diagnose type of error is avoided.

You cannot assume the same software (you get updates, and other things
change dynamically too) will always use the same setup.

> > People can take advantage of color
> > managed output already while the community figures out a good way to
> > handle measuring.  
> 
> I'm quite not sure what you mean by that, since the actual measuring (i.e.
> obtaining instrument values) is a solved problem. What has to be done
> for Wayland is installing calibration curves and profiles. As long as
> the profiling application can do that, then it can proceed with
> calibration, profiling, installing and verification.

Wayland has a different paradigm that I have been trying to explain.

You said the measuring application needs to be able to show content
unhindered by the compositor. I agree, and for that we need a Wayland
extension taking into account the special details of measuring, e.g.
how to position the color patch you want to physically measure and many
many more aspects that an application cannot control using public
unprivileged interfaces alone.

> 
> It's unnecessarily complicated (and more fraught with mistakes, see
> above) to have some sort of special calibration mode. To send device
> RGB to the display, all the profiling application has to do
> is mark the source colorspace as being the same profile as the
> corresponding output profile. To turn calibration curves off,
> all it has to do is temporarily install a profile with linear per
> channel curves. At the end of the process all it has to do is install
> the computed profile & calibration curves permanently.
> To verify calibration it just has to send device RGB values
> through the installed calibration curves. To verify profile
> conversion it just has to label its test RGB values with a known
> source space profile. More complex from a Wayland point of view
> appears to be getting control of the test window output, location
> & size (although I get the impression some of this is doable using
> xdg protocols ?)

I agree on the goals, but I disagree on how to achieve them.

XDG extensions won't help you there.

"Install a profile" into the compositor has these drawbacks the very
least:

- "installing" is a global action, hence it means the application is
  hijacking at least a whole output if not the whole compositor to
  itself, this must be separate from the general application usage as it
  needs to be privileged and exclusive

- it assumes that the compositor works in a very specific way, that may
  not match the hardware it has to work with, maybe avoiding the full
  potential of the hardware

- the profile is for an output as a whole, which means that if you
  installed a profile that includes effects from pixel formats and
  source color spaces, you cannot show other applications on the same
  output correctly

> > As an example of orthogonality, I would like to point to the new
> > touchscreen calibration tool for Weston. Commit adding the new tool
> > lists the benefits it has over the old calibration tool that was using
> > the usual input interfaces in Wayland:
> > https://gitlab.freedesktop.org/wayland/weston/commit/b79dead1dded6744d251da5357e14c83b91cef32  
> 
> Right, but that seems to be an application that works through the compositor,
> not around it ? Having a special mode to turn off color calibration curves
> isn't necessary - installing linear per channel curves is all that's needed
> and avoids extra (confusing) API's.

Yes, it works through and in cooperation with the compositor. It needs
the compositor cooperation to be able to turn off the input coordinate
transformation that the compositor normally does, and it needs to
hijack that one input device completely. There is no other interface
that would allow it to do that than the (Weston-specific) Wayland
extension specifically designed for it.

Another advantage of the explicit special interface is that if the
measurement application happens to crash, the compositor will restore
normality automatically. The same happens if the user decides to
forcefully quit the measurement, e.g. the measurement application is
misbehaving: the compositor can offer an easy exit. (Ever had a game
change the video mode on X11 and then all your desktop icons are piled
into the top-left corner, or the game crashes and leaves your monitor
in low-resolution mode where you can't even see a desktop settings
dialog in full? Automatic recovery and explicit action intents helped
also those use cases.)

Likewise, color measuring needs to make "the curves" linear like you
say, and it needs to hijack one output completely. There is no
interface that would allow the measurement application to do that, so
one needs to be designed according to the Wayland paradigm.

> > The reason the extension is specific to Weston is that there was no
> > interest in making it a standard, no-one else saw the need to calibrate
> > touchscreen input. It could still be standardised if the interest
> > arises.  
> 
> Sure, and having a defined protocol and example implementation in Weston
> is about all one can expect in the case of color management. Having
> a working example and tools and applications that have been proven to
> work correctly with the implementation provides a path for wider adoption
> when demand arises.

I think a color management related extension for content delivery should
aim higher than that from the start: to wayland-protocols as an unstable
extension. The cross-compositor and toolkit demand exists already.

An analogy to the touchscreen calibration case is wl_touch which is
the public unprivileged interface that touchscreen apps are already
using. wl_touch only happens to be in Wayland core because it was
designed early in Wayland's life. The original touchscreen calibration
app used wl_touch, and was found to be sorely lacking because of it.

> > Likewise, measuring the color characteristics of a monitor will likely
> > require a notion of intent to a Wayland compositor. Not only one wants
> > to be absolutely sure the compositor is not mangling the pixel data,  
> 
> Quite the contrary - see above. If the compositor is mangling all the
> pixel data, then it needs to mangled in exactly the same way while
> profiling so that the response is the same and so that the profile
> is valid for the compositor mangling pixel data that way. To have
> it un-mangled while profiling, yet mangled for all the applications
> means that the profile is invalid.

I believe we can do better than that.

Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20190129/825c9239/attachment-0001.sig>