[RFC wayland-protocols] Color management protocol

[Graeme Gill]

Pekka Paalanen wrote:

Hi,

> I suggest that compositors use the CMS you have spent so much time and
> effort perfecting, and you start with the assumption that they will not
> or cannot do so. Why?

We need to be clear on terminology here - "CMS" can have a few
meanings, depending on the intent of the user.

CMS in the sense of the tools I've written is a suite
of applications. Unless my understanding of what
a compositor is, is very wide of the mark, I really
don't think you want to incorporate something like
a quarter of a million lines of extra source code that
is organized into a flexible suite of inter-operating
tools that work together, into a compositor.

(CMS can also loosely refer to things like a CMM
 (Color Management Module). A CMM  typically has much better defined
 functionality and smaller scope, and can certainly be incorporated in a
 compositor - i.e. lcms).

As an application writer, I don't think incorporation
into a compositor is a reasonable proposition either - the very
purpose of an operating system is to be a good
environment to support applications, so to propose
re-implementing the applications within some sort
of quasi-non-standard operating environment
such as a compositor, has nothing attractive about it.
(Take that to the power N if there are N different compositors.)

To have the compositor call out to the tools rather
than incorporate them is perfectly possible
(Richard does some of this in colord I think), but doesn't
solve the problem of how the tools can do their job
if the graphics sub-system (i.e OS) doesn't support the facilities
they need to access the graphics hardware.

> Are you implying that the CMS you worked on so hard is impossible to use
> from a compositor?

As incorporated code - yes - I don't think it is practical.

> No, I do not mean forking the CMS and shoving the code into a compositor
> project.

> I mean libraries, services, daemons... does your CMS really not have
> any such interfaces?

No - it's an application, so it's organization is private - it doesn't
aim to provide services to anything except its own components. But even
if it was so organized, how does this solve the problem of those services
not being able to access the hardware ?
Why would the compositor or desktop environment want to re-implement
the applications UI's ?

>> Answer - this seems rather unlikely to be a viable path. It's
>> not reasonable - color management applications need
>> to be independent applications to be practical, and the
>> alternative is fairly simply by comparison :-
>> provide the API's needed as standard.
> 
> Yes! The CMS needs to provide the API that all compositors could use.

That's back to front. The CMS needs the API's that the Compositor provides,
since the Compositor is the operating system element that
is providing a standard and coherent API for applications
to access the display hardware.

> Even in the X11 case, simply shoving something into the CLUT directly
> did not work.

Terminology confusion - in ICC speak, cLUT is a multi-dimensional Color lookup
table (aka "3DLUT" in the 3D case, aka "mLUT"). The terms
I'm familiar with for the single dimensional lookup tables
in the (used to be) Video D/A are either "RAMDAC" (From the Brooktree HW) or
"VideoLUT", the latter I thought being the generally adopted term.

> You can have multiple apps messing with the CLUT (e.g.
> redshift, right?) without knowing about each other, leaving the user
> clueless of the state of the display.

There's a big difference between "did not work" and "could be
improved". In practice it all works well enough, because few applications
mess with the VideoLUTs, and the user can fix it if there is
a problem, by not installing or enabling applications that clash.
This is little different than the user dealing with other
clashes, such as which application is currently talking
to a serial port, which application is currently full screen, etc.

And "can be made to work" is infinitely better than
"impossible, because the application environment doesn't give access".

Could it be improved to better manage or resolve such clashes - yes!

But for a user who wants color management, there is no clash - they
don't run things like "redshift" or applications that alter the brightness
of the display, because that will mess up their display profile
and hence color accuracy.

> I believe modern display hardware
> start with a CLUT-matrix-CLUT pipeline and continuously invents more
> ways, so "just set the CLUT" is already an outdated approach, not to
> mention that each hardware plane might have its own pipeline setup

This is a complete distraction.

1) Saying "there's more complicated stuff that could be standardized
   in the future, so lets not implement things that people currently
   depend on now" is not serving the users at all, and could go on
   indefinitely.

2) CLUT-matrix-CLUT has no application to display calibration. It
   has possible uses I'm sure, for instance you could use it to make the display
   do a rough emulation of some other colorspace, the accuracy of which
   depends on how additive the native display response is (typically
   you can't assume more than good additivity, which is one reason why
   cLUT based ICC profiles are often used), as well as having no real
   control over gamut clipping :- but if a display is connected to
   a general purpose operating system which aims to make the
   full capabilities of the display available to applications,
   the matrix and second 1D LUTs are useless. The reason is
   that any matrix which is non-unity can only make the available
   gamut smaller, or present a gamut boundary that is unexpectedly
   small (i.e. a gamut boundary that doesn't coincide with the
   apparent device gamut boundary).

> The situation on Wayland not that different. You still need the
> { display server, window manager, compositor } process to work *with*
> CMS to produce best results, and it even offers significant benefits
> like choosing a more appropriate blending space or automatic and
> GPU-accelerated color mapping, plus preventing fighting over control.

All those things may be desirable, yes.

> You don't have X11 for communicating between the compositor and CMS,
> now you need a library interface. It's not something one wants Wayland
> for, because Wayland is IPC, implying the two parts are in separate
> processes.

Who is standardizing and providing that library interface in
a Wayland based system, so that applications can use Wayland
instead of X11 ?

Why do you think that the Compositor and CMS are not separate
processes ? It is certainly the case if the CMS is a set of independent
applications.

Why is it the compositors job to provide the API (Wayland) for accessing
just some parts of the display hardware, and not all of it that
applications need access to ?

> If you design the compositor-facing interface as a library ABI in your
> CMS, then you have the power to design it right and tell compositor
> developers how to do the right thing by using it.

If Wayland ends up being a remote protocol, how does this work
to a remote display server ?

> This, added with the power of adding implementation-specific Wayland
> extensions at runtime(*) from within the CMS, should let you implement
> much much more than you ever could if you stuck with the X11
> architecture "I need to control hardware directly from a CMS tool that
> is a Wayland client and the compositor has to stay out of the way".

Realistically, color management isn't crying out for "much much more",
just "is possible to implement current functionality" would be
a good place to start. (And I don't mean to imply by
this that things should be implemented in exactly the same
way as existing systems.) This is essentially because this is
all an application centered activity for color sensitive tasks,
so the required display server involvement is minimal.
As I understand it, Wayland already provides the ability
for applications to make use of the GPU in rendering to
their buffers, so taking advantage of this for color
management doesn't need any further enabling by the
display server. Lots of more basic things are missing though.

A display server providing some level of default
color management or basic color management for
otherwise non-color managed applications would
certainly be a good thing as well, and a reason to
incorporate at least some basic CMM capabilities in
the compositor.

Having a standardized means of coordinating the
display profile would make it possible for the compositor
to use that information to provide better blending and
anti-aliasing behavior at some cost to performance,
although color sensitive users probably won't want
to make use of transparency.

Regards,

Graeme Gill.