[RFC wayland-protocols] Color management protocol

[Daniel Stone]

Hi Graeme,

On 21 December 2016 at 02:57, Graeme Gill <graeme2 at argyllcms.com> wrote:
> Daniel Stone wrote:
>> I recommend you re-read Pekka's emails a couple of times until they
>> sink in. It is clear that you are extremely experienced in colour
>> management;
>
> I think that applies in the other direction as well.

I would definitely benefit from seeing an expansion of some of the
terminology that is thrown around which can be subtly misleading. But
yes, as said, I've got some Christmas reading to do.

>> If you reduce Wayland's capabilities to that of X11, then some of
>> your suggestions may be less wildly unsuitable, but on the other hand,
>> we might as well be using X11 if that were the case.
>
> It's not my intent to do so, but to contrast the gap in
> capabilities that currently exists, and would like
> to see filled in a workman like manner.

Per my reply to Chris, some of those gaps can be filled in ways which
are not obvious when coming from other environments. Trying to solve
them in the exact same way would be bashing a square peg into a round
hole, so it can be useful to step back and have a think about
first-order requirements, rather than just jumping directly to deeply
specific solutions.

>> Let's say that the target here is a user for whom absolute colour
>> correctness is so critical, that they have deeply specialised
>> applications and hardware, monitors most people have never heard of,
>> secondary measurement and calibration hardware, and have sunk a great
>> deal of time into specifically configuring this.
>> For such a user to pick a compositor which was actively bad at colour
>> management would be rank stupidity.
>>
>> If you don't trust the compositor you're using to do the thing which
>> is above all else most important to you, maybe pick something else?
>
> If there is no capability or standardization in this area for Wayland based
> systems, I don't see how such a user has any other choice than to pick a
> "compositor" called Microsoft Windows or Apple OS X. That would be a shame.

Yes. No-one in this thread wants to see a solution which precludes
fully correct and accurate colour management.

>> You're painting a false dichotomy where either people use external
>> tools to allow colour-critical users to compensate for the damage
>> their known-bad compositors do, or there is no chance of ever having
>> colour management ever. If only there was a third option.
>
> I'm not sure what you mean by a "known-bad compositors",
> nor do I really understand how this paints a false dichotomy.

By 'known-bad compositor', I speak of a compositor which, when
processing information from clients, does so destructively. Such that
the final output from the compositor (to any connected display, or
screenshot/screencast, be it via means of direct scanout from a plane,
or intermediate GPU rendering, or an intermediate hardware composition
through a dedicated block, or software composition, or, or), produces
incorrect results.

A lot of the discussion on this thread seems aimed at ensuring
compositors are not involved in colour management and colour-critical
applications are forced to use external systems to route around them
in order to achieve the desired colour result. This is a
self-fulfilling prophecy.

> The external tools are a means of creating the data to
> allow for managing and compensating for their particular displays.
> A competent application to create this data is non-trivial.
> If a graphical systems makes implementation of such software
> difficult or awkward or laborious, then such software
> may be slow to be developed for such systems. Without
> these tools then either color management can't exist
> on those systems, or it is compromised (i.e. using EDID
> derived profiles, or stock profiles), or awkward workarounds
> are needed (switch to an X11 server, boot Windows etc.)

Completely understandable. Again though, I think it's important to
separate the discussion of how to create a calibrator, from how to
create a colour-aware application that can achieve its desired colour
results. The two may, at a very low level, share similar requirements
(along with that of managing the data and applying it to outputs etc),
but I expect them to look very different from a protocol point of
view. Some of it may, as said earlier, not involve Wayland at all.

>> This thread perhaps demonstrates why many projects which would
>> otherwise care about colour management have such difficulty supporting
>> colour management correctly.
>
> It's a difficult subject to get a grasp on at times. Many programmers
> get to the stage of knowing what RGB is, and thinking there is not
> much more.

Yes, but it's also how you work with people. Security had the same
issue for the longest time, where there were application/system
people, and then there were security people, and the only time they
met was to tell each other that they didn't live in the real world. I
desperately want to avoid colour management remaining a ghetto where
the only way to get correct results is via complex external systems
which attempt to route around the underlying display pipeline. I want
it to be a first-class part of our system. But this thread is not a
good start.

>>> Setting VideoLUTs has been standard in display systems almost
>>> forever. Find a way of implementing support in Wayland, so
>>> that color management can happen in Wayland.
>>
>> No.
>
> I think you need to be a whole lot more flexible here,
> if progress is to be made. I'm open to other ideas on
> how to manage this, but "no, it's not possible"
> does not work towards this. Progress might be
> made if you or others with a better grasp of
> Waylan's architecture at least understand why, and what,
> rather than simply saying "No" without any understanding.

I totally understand how driving (one of) the video LUT(s) is a
reasonable way to achieve your end goal.

>> The problem here starts with 'it's not even a complete solution',
>> continues on via 'colour control within display hardware is now more
>> complex than you realise', and ends with 'this is limiting enough to
>> be the worst of all possible worlds'.
>
> I'm not sure what you are talking about.

Simply providing video LUT data and nothing else is not a complete
solution, because it means GPU-based composition pipelines (they do
exist for non-alpha-blended cases too) are unaware of colourspace, and
are thus liable to be destructive to the end result, especially if
they attempt to do things such as taking clients with no explicit
source information to be in sRGB colourspace.

That you keep saying 'program the video LUT' makes me think you
perhaps do not realise the full complexity of colour management in
modern display hardware (again, see other thread). This includes that
LUTs can sometimes be applied on a per-plane basis as well as
per-output.

The last part is important, because we can now sensibly make use of
hardware overlay planes in Wayland compositors. If we have enough
information about the colour source, we can even do it in such a way
that it still achieves perfectly colour-correct output. Treating 'the
video LUT' as a first-class, client-visible, object in and of itself
removes the compositor's flexibility to promote clients to hardware
planes, forcing us to choose between performance (a notable dent on
battery life; pushing thermals high enough to trigger limits; putting
enough extra load that full framerate cannot be achieved), and colour
correctness.

>> Reading you reiterate this yet again, I'm stunned that you want to
>> explicitly design for a system where a core and indispensible
>> component of the rendering pipeline, is incapable of correct
>> rendering.
>
> Sorry, I don't know what you mean by that. Can you be more
> explicit in what you mean by "core and indispensable
> component of the rendering pipeline", and "is incapable of
> correct rendering" ?

The compositor is a core and indispensable component of the rendering
pipeline. There is no way for Wayland clients to display anything on a
display, without the compositor mediating the display. Often the
compositor will need to perform intermediate rendering in the process.
The compositor is empowered with vastly more flexibility than previous
rigid systems, and as such is capable of a lot more than you may
realise.

If the compositor does not have fully accurate information about the
colour properties of client surfaces, it stands to reason that
anything the compositor does outside the specific case the client
targets, will not retain colour accuracy. It could attempt to
guess/infer this, but building that into system design seems
self-defeating.

>> Niels, I think conceptually you have the foundations of a good system
>> in this proposal. I need to do some more looking into colour
>> management (bit of light reading for Christmas), so hopefully I can
>> have some more pertinent questions and suggestions after that.
>
> It would be great if others did a bit of that too, so that
> there is more chance of some of things I've said being
> understood, or even taken seriously. (And yes, I will continue
> to poke away at understanding Wayland a little better.)

I don't think anyone doubts your expertise in the specific field, but
much like a compositor with no explicit information on client colour
properties, I'm having to try to walk back on your reasoning, to go
from specific solutions you appear to be demanding, back to actual
first-order problems. A lossy and frustrating exercise.

Wayland and X11 are no more similar than two surfaces both in RGB
colourspace. They may look so from afar, but at the moment it just
appears like you're trying to bash a square peg into a round hole. In
order to do this properly, and finally give colour management its
deserved first-class place in open-source window systems, you really
need to take a step back and start unpicking some things you and
Argyll have previously taken as a given. At that point we can progress
from both sides saying 'no', to constructive design discussion. I'm
incredibly grateful to Niels here for starting this particular fire,
because his proposal is a model we can actually pick at, reason about,
and make concrete suggestions to.

Cheers,
Daniel