[RFC wayland-protocols] Color management protocol

[Graeme Gill]

Carsten Haitzler (The Rasterman) wrote:
> On Sat, 17 Dec 2016 21:16:41 +1100 Graeme Gill <graeme2 at argyllcms.com> said:

>> That's not a typical situation though, but nothing special would be
>> happening - a new profile may be installed by the user as well,
>> in which case an application should re-render to accommodate
>> the change.
> 
> the user at most should be interacting with compositor settings/tools to
> configure a specific profile for a display (let's assume a fixed profile for
> that screen), so a compositor tool to tell the compositor the profile changed
> (pressed a button on the monitor to change it). when they alter the compositor,
> then compositor can tell applications.

"Compositor tool" == color management application, such as ArgyllCMS "dispwin -I"!

>>  > yes. compositors right now work in display colorspace. they do no
>>  > conversions. eventually they SHOULD to display correctly. to do so they
>>  > need a color profile for the display.
>>
>> For enhanced color management yes. But core comes first, and is necessary
>> for many color critical applications, because the compositor will never
>> have the color transformations they require.
> 
> they will have to have them, or then not support that colorspace at all.

There's nothing "have to" about it. It's technically impossible for
a compositor to satisfy every applications color management requirements,
because they could be defined by the application, if it has sufficiently
specialized needs.

And I'm not sure what you mean by "supporting a colorspace".
A source colorspace is supported if a device profile exists for it that the CMM
knows how to handle. But that is not the complete story, because there
is then the details of how that profile is to be used, i.e. the
details of how it should be linked to the output profile.

> not in wayland. not acceptable. the app will never know which windows it is on.
> as i have said - could be wrapped around a sphere or a room full of bunnies
> bouncing about across 8 screens.

It's also not acceptable that color be wrong. So how about trying to
come up with a solution, rather than saying "Wayland hasn't been
designed for that requirement up to now, so it can't be done" ?

> it can't if it doesn't know how a window or buffer is transformed or mapped to
> screens.

So provide mechanism for it to know!

>> The code is already there to do all that in color critical application.
> 
> then they get to pick a colorspace and render to that.

There is no colorspace to pick - a display has a characteristic
behavior. There is no choice in it, short of tweaking it's controls,
altering its calibration, or changing one of its settings.

> attahc that to the
> buffer.

Different display devices have different characteristics, hence
different profiles. So one choice will not work for a surface that
spans multiple displays.

> compositor will do whatever it wants. e.g. if that buffer is on the
> screen it matches it'd do no transform. maybe it does no transforms at all. and
> simply hovers some indicator above the serface telling you if the surface
> colorspace matches the screens colorspace or not. maybe compositor disallows
> the window to be moved off the screen that matches the colorspace. that can be
> configured via the compositor.

The user will want to configure their windows as they see fit, including
moving images from one display to the other. There needs to be a mechanism
to allow color accurate display when this happens.

> and now you just hit "unacceptable in wayland land" space. which is why i tell
> you that this is not acceptable.

Wrong color is unacceptable too. So how do you solve it ?
(And it's a solved problem on all other systems. Why do
you want to cripple Wayland ?)

>> It's damage, just like any other, and color critical users using
>> color critical applications will take "trails" over wrong color
>> anytime. No "trails" and wrong color = a system they can't use.
> 
> and totally unacceptable for wayland space. so a big fat no to that kind of
> design.

So Wayland is a big fat no for anything that requires accurate color ?
Wow. And I though Wayland was intended to ultimately replace X11.
Apparently not!

>> Yes - exactly what I'm suggesting as core color management support.
> 
> but it doesnt have to tell you which screen it is nor tell you which screen
> your window is on. it's an option of 1 of various colorspaces to be able to use
> given a specific compositor.

I don't see how this can work, except for limited color accuracy requirements.
Pick a small colorspace with a common gamut, and the full gamut of all displays
can't be used. Pick a large gamut and clipping policy can't be managed.

>> "Every pixel being perfect" except they are the wrong color, isn't perfect.
> 
> IF the compositor is doing the transformce of colorspaces, then you can acheive
> perfect pixels.

No you can't, if the compositor doesn't have the information or algorithms
to do the transformation. This is absolutely no different from any
other situation - the compositor can't render stuff it doesn't have -
ultimately the application has to provide the content that appears on
screen, and the compositor is its agent in doing so. Color is part of
the content.

> reality is compositor do have to do transforms to get 601,709 and the new 2020
> colorspaces right for video anyway.

What's that got to do with anything ? They are just colorspaces, like any
number of other RGB colorspaces. (And yes, and I released a public domain
ICC profile for Rec 2020 in 2013).

>> That's the reality of how displays work. The user presses a button on the
>> front that says "emulate sRGB" or "native" or "Preset 1" or something else.
> 
> i mean control by applications. by wayland clients.

That's pretty rare - most displays use poorly documented or standardized
means of setting such options programmatically (EDID data lines or some have a USB).

>> And color critical users will scream bloody murder at anything related
>> to color that isn't under their control, if it affects the accuracy or scope
>> of the color workflow.
> 
> that's what compositor tools are for. the compositor settings are going to be
> how you load/configure color profiles for a display. not via protocol.

Huh ? How are "compositor" tools meant to be written ? They need a means
of talking Wayland to do so !

> of course they are not exact. they are approximately srgb which is vastly
> different from adobe or wide gammut etc. no display other than insanely priced
> professional displays is going to be EXACTLY sRGB or adobe rgb etc. it's going
> to be a bit off and thus some color profile adjusting via either some table/lut
> with interpolation is going to be needed.

Right. Color management is needed! That's the topic at hand.

>> That's up to the user. The user may have something else they can
>> assign if they are unable to profile the display (EDID derived
>> profile, model generic profile, etc.)
> 
> that's the compositor that deals with this internally. via its own tools and
> settings. not some generic user tools or photoshop app etc.

What are "its own tools" ? You mean color management tools - the ones
used to calibrate, profile and configure the display! The ones I'd
like to be able to port to Wayland, if it had the actual protocol/API
support needed to do so !?

>> It's not about bit depth, it's about algorithms. No compositor can
>> do a transformation that it doesn't have an algorithm for.
> 
> and then it doesnt offer it if its display methodology is to transofrm to get
> colors uniform. pretty simple.

So you basically saying that Wayland applications will have to have
color management that's crippled to the compositors color transformation
capability, even if the application itself knows exactly what to do ?

>> As I've explained a few times, and extension is needed to provide
>> the Output region information for each surface, as well as each
>> outputs color profile, as well as be able to set each Outputs
>> per channel VideoLUT tables for calibration.
> 
> that's not going to happen. it's a wayland design premise that applications
> should not know this.

So you're condemning Wayland to be unsuitable for color critical
applications and users ?

>> Let me give you an example. The application has a ProPhotoRGB buffer,
>> and wants to render it with image specific gamut mapping into the display
>> space. It has code and algorithms to 1) Gather the image gamut, 2) Compute
>> a gamut mapping from the image gamut to the Output Display gamut, invert
>> the A2B cLUT tables of the Output display profile to floating point
>> precision with gamut clipping performed in a specially weighted CIECAM02
>> space.
>>
>> I'm not quite sure how the Wayland compositor is going to manage all that,
>> especially given that the application could tweak or change this in
>> every release.
> 
> app provides the above mapped data.

Huh ? You've just finished telling me that that can't be supported
by Wayland !

> it ends up being an R, G and B value for
> one of the compositor supported colorspaces - e.g. one of them is monitor #3.
> when on monitor #3 the compositor does passthrough of that buffer. it displays
> with no changes (for example). but when it is on another screen the compositor
> maps FROm the colorspace of that monitor TO another. yes. this can b e lossy.

Not lossy - inaccurate, and not according to what the user is expecting.

> you can dither of course (and even do temporaly dithering) to approximate
> colors. it will look best on that display. it'll look "as good as possible" on
> other displays.

That's not acceptable when color accuracy is expected. I was really
hoping that every pixel would be perfect, not compromised due to
the inability of Wayland to accommodate the application providing
the correct value.

>> A pixel is not perfect if it is the wrong color.
> 
> flickering/trails is worse.

A user who is doing color critical work will disagree with you, and
they will be the ultimate arbiter of what is a viable system. They
care a lot less about such aesthetics, if it has to be a trade off
against color accuracy. (And I'm not convinced that there has
to be such a trade off).

>> A color profile can be quite complex, including scripted in the case of
>> something like an OCIO or ACES profile 
>> <http://www.oscars.org/science-technology/sci-tech-projects/aces>.
>>
>> But the device profile is only half the story - it does nothing on its
>> own, it needs to be linked with another device profile. And the flexibility
>> at that point is unlimited.
> 
> and the device end is all a compositor cares about.

Huh ? You've just spent considerable energy arguing that the compositor
has to the conversion to device space, because only it is allowed
to know how a surface maps to Outputs!

>> Bad way of doing it, for reasons I've pointed out multiple times.
>> Be explicit rather than rely on a trick - use a switch.
> 
> it's not a trick. its a standard null op. src fmt == dest fmt. no conversion
> needed.

It's not naturally a no-op. Pixel values can change if you actually
perform such a conversion.

>> Sure, but that's not an aspect I've mentioned. Ultimately the display
>> is RGB, irrespective of the encoding using to carry that information
>> to it.
> 
> without a colorspace attached to yuv buffers you cannto know how to transform
> them to rgb (srgb) values correctly at all.

> ignoring screen output colorspce
> differences entirely here. just plain math to srgb space.

Sure, you need to know the decoding transform from YCrCr to RGB if that is
your workflow. But that conversion doesn't necessarily define the colorspace,
since the underlying RGB may or may not correspond to a standard colorspace.
It may be a device profile RGB instead, for instance.

>> Sure - complexity in managing encodings. But that has nothing
>> directly to do with color management, which is about colorspace
>> differences.
> 
> its the same thing - it's mapping a set of colors to another set in order to
> get correct physical colors.

No it's not. An encoding change is just a mathematically different
representation of the same colorspace. There are mathematically
exact reversible equations for the transformation, but the
underlying colorimetry is defined to be the same. (i.e.
the primaries of a YCbCr space are that of the RGB primaries
it was derived from).

>> To be fare, I'm not that aware of how the hardware presents itself
>> in regard to such things (data sheets seem hard to come by, and I
>> have gone looking for them in vain on a few occasions), but for many color
>> critical uses, it's not an immediate concern because such applications
>> are not going to be using yuv buffers. (Exception might be a video
>> editing/color grading application sending previews to a TV or
>> studio monitor - but all that is about encodings rather than
>> colorspaces.)
> 
> tell the people building millions of smart tv's that yuv color correctness
> doesn't matter for that tv they are selling for $20,000+. :) hell even over
> $100k. yuv color correctness matters just as much.

You've misinterpreted what I've said. I said the encodings
don't matter, because it's the underlying colorspaces that matter
with regard to color accuracy.

[ Are you really unaware that people use my color management
  tools to create workflows that allow accurate emulation
  of Video display standards ?

<http://www.avsforum.com/forum/139-display-calibration/1464890-eecolor-processor-argyllcms.html>
  <http://www.avsforum.com/forum/139-display-calibration/1471169-madvr-argyllcms.html>
  i.e. I have a clue.
]

> the compositor will have configuration/tools/data.

No it won't, if there are no Wayland protocols & API's to make that
possible!

>>  > dumb compositor example:
>>  >
>>  > compositor reports 2 colorspaces:
>>  >   null transform RGB
>>  >   BT.709 YUV
>>
>> Why is it reporting an encoding rather than a colorspace,
>> and why isn't it providing the two display profiles ?
> 
> it doesnt know them from a stick in the mud. it's a dumb compositor.

Huh ? This is what color management extensions provide !

>>  > smart compositor:
>>  >
>>  > compositor reports 5 colorspaces:
>>  >   null transform RGB
>>  >   wide gammut RGB
>>  >   sRGB
>>  >   BT.709 YUV
>>  >   BT.601 YUV
>>
>> I don't see how extra encodings are useful without their
>> corresponding color spaces.
> 
> as i have said before. the colorspace is a color transform WITH adjusting
> constants/matrix/LUT etc.

How can that be done without the colorspace profiles ?

>> In the dumb or core case, it has two display profiles, one
>> for the professional grade monitor with wide gammut rgb output,
> 
> no. the dumb case is "compositor has no idea. so i do rgb and yuv". it has no
> clue otherwise.

That is the point. The user can use their color management tools
to profile their displays, register the ICC profiles with the
compositor, and then the color-aware application can then fetch the
output profiles for the display, and color manage the pixel
data given to the compositor. So color management works.
Existing color sensitive applications have all the code
to do that.

>> and the other for the bargain basement bin display from walmart.
>> It can then transform source images in whatever colorspace they
>> are tagged with, into the appropriate display colorspace,
>> in the way the application and user needs it to be transformed.
> 
> it has no clue what the bargain basement bin monitor displays. the dumb
> compositor isn't even going to try.

Nor should it for core support. But that doesn't stop
color management if the application is given the opportunity.

>> If this is really the case, then the conclusion is that Wayland is
>> not suitable for serious applications, and certainly is not a replacement
>> for X11. I don't actually think that that is true.
> 
> multiple people have told you this by now. applications are not going to know
> what screen(s) their buffers span. it's abstract. equating this to "then
> wayland is not for serious use" is sticking your head in the sand because
> wayland does not work like every other display system that is 20 years old.

So rather than finding reasons why it can't be done, how about
finding ways that it can be done ? Is the development
of Wayland really so inflexible and out of touch, that
when faced with the problem that one of the assumptions
made is incorrect, it can't adapt ?

>> How can there be any color management unless the
>> colorspaces of each display is recorded somewhere ???
> 
> as i have said several times. colorspace includes the details of the mapping.
> e.g. a LUT or matrix or whatever blob sufficiently describes the mapping.

And as I've pointed out several times now, this isn't sufficient if
the source profile can't be described that way (ACES etc.), or
if the actual use of the profile (linking) requires more nuance
that the compositor knows how to do.

>> Because that's not actually true. Each real display has it's own
>> response. That's why I write tools to profile displays, and why
>> people use those tools.
> 
> yes. i know its slightly off and correcting input can make it look right (to
> some extent).

It's not slightly off - people spend a lot of time and money
making it right.

> i'm talking braodly that most displays are nominally sRGB, or
> nominally adobe rgb, or something else.

Oh boy. Yes, if you don't care much about color. Definitely not if you are
doing something color critical, like photography, desktop publishing,
color grading, etc.

> there is a small bit of adjusting to do
> to make them "close to perfect" and thats where your colorimiter plus doing and
> actual profile of the display and recording the data helps. or maybe there is a
> manufacturer provided file for that already...

All I can suggest is that you become a bit more familiar with
what color management entails. There is a lot of info out
there on the web. My small contribution is this:
<http://www.argyllcms.com/doc/ColorManagement.html>

Summary: there is a lot more going on that "a little bit of adjusting".

> no out-of-compositor tool will control those curves.

I hope your wrong. Because no in-compositor tool has the smarts
needed to take on color management functions such as calibration
or profiling. Without color management tools having access to the curves,
I can state categorically that Wayland is unsuitable for color critical
applications, because it can't be calibrated. Shall I start spreading
the word ?

> likely they will provide
> some calibration file that the compositor consumes and either adjust output or
> monitor to correct it, or transforms pixel data to correct, OR it can pass that
> data back to clients via the colorspace list.

Completely inadequate. If you don't really know very much about
color management, then you aren't in a position to make such
value judgments on the users behalf.

>> No it's not - it already works on X11/OS X/MSWin.
> 
> not in wayland. bunny rabbits.

So when will there be a replacement for X11, if Wayland isn't it ?

> which is just a list of colorspaces (as i've sayd - they contain transofmr data
> such as matrices, lut tables etc.).

I think I have a clue what an ICC profile contains.

>> It can't be out of scope if the compositor is to do color management.
> 
> compositor adjusts from an output colorspace to another. source is not
> important to it. it also may not adjust/map and just add some indicator.

Output to output transforms will make any but a single
display 2nd class. This is hardly "every pixel is perfect".

> no - i'm talking the device rgb colorspaces (yuv happens to be included too as
> video colorspace definitions with yuv values map to specific defined rgb
> output).

OK - that was your out.

>> The whole point is that each display has a different color response,
>> and incoming color should be transformed to compensate for these
>> differences. So each display (ideally) should have an associated
>> color profile.
> 
> but clients shouldnt know which output it's for or which output they are on.

You say that, but you can't back it up. I've explained at length and
rather repetitively why it is necessary, and you haven't
offered any technically feasible alternatives.

I'd be really interested to learn if everyone else working on Wayland
really thinks that accurate color doesn't matter.

> i'm not talking close. i'm talking "i used colorspace 36 in your list for this
> buffer". and colorspace 36 is the exact profile of monitor B. and the window
> now is displaying ONLY on monitor B. no transform needed at all for display
> there.

OK - if there is a level of indirection (as suggested by Niels Ole Salscheider's
extension suggestion), then this is not ambiguous. The issue of
identifying which output profile is the one that a surface is going
to land on, remains. Without this the client can't know which
output profile enum to assign it to result in a null transform.
(And the bootstrap issue remains.)

>> That's what a null transform is though - a forward conversion
>> (Device space to PCS) followed by an inverse transform
>> (PCS to Device space).
> 
> which is inputpixel == outputpixel. nothing changes. no transform is done at
> all.

Not true. For instance:

xicclu -ff -ir display_lut.icm
1.000000 1.000000 0.000000 [RGB] -> Lut -> 97.741833 -15.695131 91.298053 [Lab]

xicclu -fb -ir display_lut.icm
97.741833 -15.695131 91.298053 [Lab] -> Lut -> 0.991896 0.992829 0.084499 [RGB]

>> That's not possible if there was no previous execution.
> 
> it could have a database of every monitor in the world and edid matches and get
> it right first go. for embedded use the device maker would pre-configure the
> device with the correct profile at the factory.

Right, but you're performing contortions to avoid just saying what the
situation actually is - that there is no profile.

> yes. and profile can be pre-shipped, some database etc.

So you're assuming that people can only use systems that are
pre-packaged ? How do the packagers set them up initially ?
(i.e. there is a cascade of issues just to avoid adding one flag that
 says what the situation actually is. This has flow on effects - for instance
 if there is a flag, then the application can tell the user "this display
 doesn't currently have a profile - the color is probably incorrect", rather
 that not knowing what the situation is).

> as long as it's not necessary to support it and applications run even if their
> colors are a bit off.

"Core" level is mainly information - the application is able to
use it if it chooses, and fall back to no color management,
or even fail if the author decides that is the best.
"Enhanced" level is a convenience for an application,
and a tool for the user to manage non color aware applications.
An application has similar choices to above - fall back
to core, fall back to none, or fail if the author decides
that the application has no value without color management.

Graeme Gill.