[RFC wayland-protocols] Color management protocol

[Graeme Gill]

Daniel Stone wrote:

Hi Daniel,

> I would definitely benefit from seeing an expansion of some of the
> terminology that is thrown around which can be subtly misleading. But
> yes, as said, I've got some Christmas reading to do.

If anybody would like some concise explanations for color management
terms and concepts, I'm willing to so explain, as well as
provide references for more expansion or detail.
Being lucid enough to succeed in conveying comprehension
is another thing of course.

> Per my reply to Chris, some of those gaps can be filled in ways which
> are not obvious when coming from other environments. Trying to solve
> them in the exact same way would be bashing a square peg into a round
> hole, so it can be useful to step back and have a think about
> first-order requirements, rather than just jumping directly to deeply
> specific solutions.

Agreed - and this is something I'm trying to do as well, to outline
the big picture as the background that leads to specific suggestions.

>> I'm not sure what you mean by a "known-bad compositors",
>> nor do I really understand how this paints a false dichotomy.
> 
> By 'known-bad compositor', I speak of a compositor which, when
> processing information from clients, does so destructively. Such that
> the final output from the compositor (to any connected display, or
> screenshot/screencast, be it via means of direct scanout from a plane,
> or intermediate GPU rendering, or an intermediate hardware composition
> through a dedicated block, or software composition, or, or), produces
> incorrect results.

Incorrect in the sense of spatial rendering, or color values/fidelity or both ?

> A lot of the discussion on this thread seems aimed at ensuring
> compositors are not involved in colour management and colour-critical
> applications are forced to use external systems to route around them
> in order to achieve the desired colour result. This is a
> self-fulfilling prophecy.

That's not what I'm suggesting. Compositors have a vital
role in allowing different applications to share a desktop,
as well as help applications and toolkits compose and
operate a GUI. But that is all about buffer management,
spatial rendering and coordination of updates. Transparency
and color encoding formats starts to dip into some color
related aspects, but only in a simple way that isn't related
to the colorimetry. A color managed workflow is about how
color is best preserved between different device dependent
color spaces, and in the context of a display system driven
by applications, this is primarily about the color spaces the
application takes in, or defines its own colors in, and the
display colorspace. (And lets be clear, when I say "colorspace"
here I'm speaking of the colorimetry, tonal response and color
mixing characteristics represented by the (often) RGB values,
although applications in general will deal with many other
types of input colorspaces such as CMYK, device independent
colorspaces, multi-channel spaces, etc. etc.)

> Yes, but it's also how you work with people. Security had the same
> issue for the longest time, where there were application/system
> people, and then there were security people, and the only time they
> met was to tell each other that they didn't live in the real world. I
> desperately want to avoid colour management remaining a ghetto where
> the only way to get correct results is via complex external systems
> which attempt to route around the underlying display pipeline. I want
> it to be a first-class part of our system. But this thread is not a
> good start.

Agreed it's not a good start, and I guess the gulf in
understanding is far wider than I have imagined from my
normal technical interactions with people.

> Simply providing video LUT data and nothing else is not a complete
> solution, because it means GPU-based composition pipelines (they do
> exist for non-alpha-blended cases too) are unaware of colourspace, and
> are thus liable to be destructive to the end result, especially if
> they attempt to do things such as taking clients with no explicit
> source information to be in sRGB colourspace.

I'm not sure what you are alluding to. A compositor pipeline
that alters color values in arbitrary ways will of course
make a color managed workflow difficult to impossible,
but what sort of compositor processing do you anticipate will
do this kind of thing ?

> That you keep saying 'program the video LUT' makes me think you
> perhaps do not realise the full complexity of colour management in
> modern display hardware (again, see other thread). This includes that
> LUTs can sometimes be applied on a per-plane basis as well as
> per-output.

Right - so this is perhaps something that might be made more
flexible. Currently it's kind of random whether things like video
overlay planes get VideoLUT curves applied to them or not, depending
on the hardware capability and/or diligence or understanding of the
driver writers.

> The last part is important, because we can now sensibly make use of
> hardware overlay planes in Wayland compositors. If we have enough
> information about the colour source, we can even do it in such a way
> that it still achieves perfectly colour-correct output. Treating 'the
> video LUT' as a first-class, client-visible, object in and of itself
> removes the compositor's flexibility to promote clients to hardware
> planes, forcing us to choose between performance (a notable dent on
> battery life; pushing thermals high enough to trigger limits; putting
> enough extra load that full framerate cannot be achieved), and colour
> correctness.

A straightforward solution is to provide a virtual color management
VideoLUT that then gets translated into a device dependent manner
to the actual hardware (i.e. duplication/adaptation to other planes).
Define it's purpose clearly, and it will be ahead of the current
implementations that tend towards just exposing the hardware capability
(although it would reduce to that in the simplest case).
The color workflow then retains a simple element with a clear purpose
and role, which is supported by existing tools.

Accommodating specific hardware capabilities such as video decoding
might well need something more, but since this is an area with no
current tools or API's and something that would (for good color
management) build on the conventional color workflow, this
seems like a project that would come a little later.

>>> Reading you reiterate this yet again, I'm stunned that you want to
>>> explicitly design for a system where a core and indispensible
>>> component of the rendering pipeline, is incapable of correct
>>> rendering.
>>
>> Sorry, I don't know what you mean by that. Can you be more
>> explicit in what you mean by "core and indispensable
>> component of the rendering pipeline", and "is incapable of
>> correct rendering" ?
> 
> The compositor is a core and indispensable component of the rendering
> pipeline. There is no way for Wayland clients to display anything on a
> display, without the compositor mediating the display.

Sure.

> Often the
> compositor will need to perform intermediate rendering in the process.

What sort of rendering though ?

> The compositor is empowered with vastly more flexibility than previous
> rigid systems, and as such is capable of a lot more than you may
> realise.

Maybe, and maybe not. We'll see.

> If the compositor does not have fully accurate information about the
> colour properties of client surfaces, it stands to reason that
> anything the compositor does outside the specific case the client
> targets, will not retain colour accuracy.

This doesn't follow, unless there is a reason for the compositor
to do a color transformative operation. Nothing involving spatial
transformations or even transparency really falls into that category.
(Linear light blending/anti-aliasing is a nice to have that doesn't
 need accurate colorspace information to be largely effective.)

> I don't think anyone doubts your expertise in the specific field, but
> much like a compositor with no explicit information on client colour
> properties, I'm having to try to walk back on your reasoning, to go
> from specific solutions you appear to be demanding, back to actual
> first-order problems. A lossy and frustrating exercise.

On the contrary, I think I've given copious levels of "big picture" to
back up specifics. Perhaps the overall context means that I'm not being
understood.

Regards,
	Graeme Gill.