[RFC wayland-protocols] Color management protocol

[Carsten Haitzler (The Rasterman)]

On Wed, 14 Dec 2016 23:23:59 +1100 Graeme Gill <graeme2 at argyllcms.com> said:

> Carsten Haitzler (The Rasterman) wrote:
> > On Tue, 13 Dec 2016 17:14:21 +1100 Graeme Gill <graeme2 at argyllcms.com> said:
> 
> > this == support for color correction protocol AND actually the support for
> > providing the real colorspace of the monitor, providing non SRGB pixel data
> > by clients in another colorspace (eg adobe) and it MUST work or apps will
> > literally fall over.
> 
> That's not a scheme I'm recommending.

ok. that's fine. i'm happy with that.

...snip...

> > not a format. a colorspce. R numbers are still R, as are G and B. it's just
> > that they point to different "real life spectrum" colors and so they need
> > to be transformed from one colorspace (sRGB to adobe RGB or adobe to sRGB).
> 
> My point stands. I've not mentioned new colorspaces.

if it's RGB or YUV (YCbCr) it's the same thing. just vastly different color
mechanisms. color correction in RGB space is actually the same as in YUV. it's
different spectrum points in space that the primaries point to.

 color management require introducing such things. BT.601, BT.709, BT.2020.
the compositor MUST KNOW which colorspace the YUV data uses to get it correct.
i'm literally starting at datasheets of some hardware and you have to tell it
to use BT.601 or 709 equation when dealing with YUV. otherwise the video data
will look wrong. colors will be off. in fact BT.709 == sRGB.

now here comes the problem... each hardware plane yuv may be assigned to MAY
have a different colorspace. they also then get affected by the color
reproduction of the screen at the other end.

you HAVE to provide the colorspace information so the compositor CAN assign you
to the correct hw plane OR configure the plane correctly OR configure the
yuv->rgb conversion hardware to be correct.

this is no different to RGB space but you don't tend to find hardware
specifically helping you out (yes gpu + shader can do a transform fast. i'm
talking hw layers or dedicated acceleration units that have existed for yuv
for a long time).

any list of colorspaces IMHO should also include the yuv colorspaces where/if
possible. if a colorspace is not supported by the compositor then the appjust
needs to take a "best effort". the default colorspace today could be considered
BT.709/sRGB. also you could say "it's null transform" colorspace. i.e. you know
nothing so don't try colorcorrect.

> > i really do not think this is needed. simply a list of available colorspaces
> > would be sufficient. application then provide data in the colorspace of it's
> > choice given what is supported by the compositor and the input data it
> > has...
> 
> And I think it is core, for all the reasons I've listed.
> 
> >>  enhanced: The compositor is provided with source colorspace
> >>   profiles by the application.
> > 
> > i again don't see why this is needed.
> 
> Hmm. How else does the compositor know how to transform from
> the given colorspace to the display colorspace ?

my point was i don't think it's needed to split this up.

compositor lists available colorspaces. a list of 1 sRGB or null-transform or
adobe-rgb(with transform matrix), wide-gammut, etc. means thathat is the one
and only output supported.

> > again - we're just arguing who does the transform.
> 
> It's not "just", it's the point.

not as i see it. given a choice of output colorspaces the client can choose to
do its own conversion, OR if it's colorspace of preference is supported by the
compositor then choose to pass the data in that colorspace to the compositor
and have the compositor do it.

> > i don't see the point. the
> > compositor will have a list of colorspaces it can display (either A screen
> > can display this OR can be configured to display in this colorspace, ... OR
> > the compositor can software transform pixels in this colorspace to whatever
> > is necessary to display correctly).
> 
> I don't know how many different ways I can explain the same thing. The
> compositor can't know how to transform color in all the ways an application
> may need to transform color. I think it is unlikely for instance, that you are
> proposing that the compositor support full N-Color, device links, ICC-Max
> support, etc., or the infinity of ways that haven't been invented yet to
> transform between color spaces. So the core color management requirement
> is that the application be able to transform into the device colorspace
> itself.

*sigh* and THAT IS WHY i keep saying that the client can choose to do it's own!
BUT this is not going to be perfect across multiple screens unless all screens
share the same colorspace/profile. let's say:

1 screen is a professional grade monitor with wide gammut rgb output.
1 screen is a $50 thing i picked up from the bargain basement bin at walmart.

dumb compositor example:

compositor reports 2 colorspaces:
  null transform RGB
  BT.709 YUV

smart compositor:

compositor reports 5 colorspaces:
  null transform RGB
  wide gammut RGB
  sRGB
  BT.709 YUV
  BT.601 YUV

in the dumb case your app can't do much. the smart case means that pixels
displaying on the pro montior either with null transform OR with wide gammut
colorspace get no transform done. pixels in sRGB, BT.709 and BT.601 have to be
transformed to the wide gammut rgb colorspace by the compositor. of course the
user would place the window on the best quality screen. within the color
spectrum the screens share colors SHOULD look identical.the client KNOWS the
colorspace being used and can transform/render data accordingly.

the user gets the exact effect you want - "perfect client-side handled colors"
if the app chooses wide gammut. this has the transform matrix/etc data needed
to do the client-side transforms. now if the compositor chooses NOT to
transform the data when the buffer is on other screens then the colors will
only look correct when placed on the professional monitor.

you CAN'T go rendering your buffer content differently based on screen
colorspace/profile. you may span multiple screens. your window may be rotated
at 37 degrees and perhaps be wrapped around 17 3d bunnyrabbits rotating around
ins space across 16 different screens.

the point of wayland is "every frame is perfect". you you want clients to
rendering their content differently based on what screen their window is on
then a compositor can NEVER get this right no matter how hared they try because
clients are fighting them and making assumptions they absolutely should not. i
already told you of more realistic cases of windows in miniature in pagers that
are not on the same screen as the full sized window (as opposed to the silly
bunnyrabit example above, but it's meant to make a point).

you HAVE to abstract/hide this kind of information to ALLOW the compositor to
get things right.

that is why i think the clients should not know what colorspace a specific
screen has at all. they should get an abstract list and the compositor "does
its best" (which may be nothing at all. it may be disallowing the window with a
buffer of that colorspace to leave the native screen that can display it, or to
emulate the colors by remapping etc.).

you get what you want - perfect colors ABLE to be managed/rendered/calculated
by the client using any algorithm/software/toolkit they like. they KNOW a
specific target colorspace and have to hope the compositor does the right thing.

> > the client simply chooses what colorspace to provide buffers in. it chooses
> > the one that is best for it.
> 
> A core requirement is that the client be able to know what output device
> colorspace each pixel is destined for, and provide a buffer that the
> compositor doesn't touch (i.e. it has no source colorspace label
> other than don't touch these pixels").

see above.

> > a colorspace that is enabled is when the display output for that screen maps
> > RGB values directly to that given colorspace. i.e. the common default is
> > sRGB.
> 
> No. No color transformation by the compositor, so no colorspace.

see above.

> > the display may be also able to switch to adobe rgb at the flip of a switch
> > or by request from the host system (via data lines). it may be fixed and
> > only one colorspace is every able to be displayed.
> 
> A color critical user won't put up with such things - they expect to
> be in control over what's happening, and if a system has proper
> color management (core + enhanced), there is absolutely no
> reason for them to run the display in anything other than it's native gamut.

a user actually should not have to deal with most of these issues at all. even
a color critical one. they likely shouldn't have to remember which one of their
16 screens has the best colorspace support for that image. the client can
choose to switch frame by frame from adobe rgb to srgb to wide gammut rub or
anything else. the compositor can then do whatever it thinks is best. just
because a compositor does things does not mean a user is not in control. the
compositor can provide such control to the user. just like photoshop can.

> > then it's a list of 1 colorspace: "sRGB" :) wehich is the current
> > default/state of play in wayland anyway.
> 
> No, it's a list of N output display colorspaces, one for each display.

see above. it should not be per display.

> > then you need no colormanagement at all.
> 
> As explained, yes, core color management needs support -
> control over VideoLUT state, plus registration of the output
> display colorspaces + knowledge of which output the different
> parts of a surface map to.

as you describe "core color management" - it's not control. that's simple
passive reading of the state and providing to the client. control is when you
start determining the state of these.

> > just assume sRGB (gamma2.2 or
> > whatever). zero colormanagement protocol or support needed. totally up to
> > client. the compositor MIGHT know its colorspace/profile and report it here
> > without sRGB listed then. thats how you know you only have 1 colorspace.
> 
> I'm not sure _what_ you are talking about. sRGB doesn't come into it.

sRGB is the colorspace of every HD display (or should be). how does it not come
into it?

> > how we're talking about color profiling tools that want to create a color
> > mapping/profile for that display so colors display correctly. you need a
> > colorimiter to measure output of course to do this.
> 
> Yes, and ?

you don't need anything special for color calibration beyond a null transform
and a compositor that won't go ignoring that null transform anyway for the
purpose of color calibration (when used by a calibration app).

> > this could be nothing more than a "1:1 do nothing" colorspace. that means
> > R,G and B values are transmitted with no changes. changing the content of
> > your buffer sends different signals that the colorimiter can read and thus
> > be used to create a color profile/mapping for your display.
> 
> Right - the sort of thing that is provided by a core/basic color
> management support.

and it's nothing but yet another colorspace. think of it as the identity
matrix. nothing happens. it still is a listable colorspace.

> > i don't get why you need so much complexity.
> 
> It's the simplest possible support, (hence calling it "core").
> It's needed internally anyway for a compositor to implement CMM
> operations for "enhance" color management.

it's also broken when you attach the color profile to a specific output. see
above.

> > ummm i'm talking about the values of pixels. RGB. what physical color does
> > #ff0000 produce? #00ff00 or #0000ff or #880000 or #008800 etc. etc. - that
> > #is
> > what colormanagement is all about.
> 
> No it's not. That's just one aspect of it. The main game is in
> how one device colorspace is transformed into another.

that's out of scope for wayland. HOW it is transformed is either done
client-side to present whatever source data in a given output colorspace to the
compositor OR it's done by the compositor to fix colorspaces provided by
clients to display as correctly as possible on a given screen + hardware.

> > it's about knowing the physical wavelengths
> > of light produced when those values are displayed by a screen and then
> > ADJUSTING the values provided in order to display the correct physical
> > color.
> 
> Hmm. Not really. Mostly a lot of other stuff has to go on top of that
> to make things turn out how people expect (source colorspace definition,
> white point mapping, gamut clipping or mapping, black point mapping etc.)

source definition is out of scope. that's up the the app (e.g. photoshop). the
colorspce defintition indeed covers what you say. and it is about adjusting. i
was saying the exact same thing. i am not unfamiliar with colorspaces, color
correction and mapping. it's necessary for YUV->RGB and is fundamentally the
same as RGB->RGB

> > a list of "just 1 colorspace" matches your core concept.
> 
> No. See above, and my original sketch.

1 colorspace which is the screen's output space is NOT the same? is that not
the same as a single screen system with the display colorspace on that 1
screen? how is it not the same? it's 1 colorspace exposed by compositor to
client in both cases. the SAME colorspace. how is this not the same?

the difference is that i dont think it should be per monitor. see above.
bunnies.

> > the compositor MAY NOT transform. it doesnt have to. it can lie. as long as
> > it knows the colorspace/profile of the output and reports that as the one
> > and ONLY colorspace supported in the list, then you have what you want.
> > right?
> 
> It's a poor approach to rely on the "null transform" hack. It's clumsy,
> doesn't convey the actual intent and expectation of operation, and leads
> to complications. For instance:

and that is why when a compositor DOES know the display colorspace it would
list that likely in addition to a null transform (there is basically no
downside to listing a null transform. it's the compositor just doing nothing
which is about as efficient as it gets). so other than the null transform list
the 1 colorspace which is what the screen supports - e.g. adobe rgb.

> * Null transform is a hack. Only under particular conditions
>   (matrix profile with equation defined per channel curves)

null is not a hack. it's useful for everyone who just doesn't care, and useful
for color calibration software to be able to display values on the screen
without any transform being done.

if the colorspace of a provided buffer == colorspace of output then it IS
effectively a null transform for the compositor and it does (or should do) just
that. thus client explicitly states what colorspace it wants for its buffer
after having queried the list of available ones.

>   is a profile "exactly" invertible (i.e. to floating point
>   precision). Use a LUT for the per channel curves (such as the
>   original sRGB profile), and it's not quite perfectly invertible
>   (although it may be to low precision). Use cLUT based profiles,
>   and it certainly isn't. So it has to be declared to be
>   a special case and assumed to be a null transform.

no one is asking anyone to transform anything (thus invert or anything else)
with a null transform. and if colorspaces match no one is converting anything
either.

> * Deciding what does and doesn't correspond to a null transform
>   needs to be decided. Same exact profile ? Same tags ? Same
>   description string ? What ?

"colorspaces match". to me that means either a strictly standards defined
colorspace with fixed constants and both sides agree to use it, or its
something where the constants have adjustments based on doing a color profile
of the screen. in BOTH cases i argue that if you flatten the data into some
memory blob memcmp() == 0 if they match. the best way i see is the compositor
provides a list, client chooses and just says "i used the colorspace #6 you
told me". then it does match when on display/hardware that really exactly
physically matches. if it doesn't match compositor will have to "choose what to
do". see above.

> * If a surface straddles two displays, then labeling all the pixels
>   with one of the two displays profile is not the same as not
>   touching the pixels.

either way if the client is colorcorrecting itself based on the display output
it thinks it might be on (and it may be on many display outputs or wrapped
around bunnies)... then it WILL look incorrect on at LEAST one of those
displays at some point. and the point is to not look incorrect.

> * What happens at startup, before the output display profiles are
>   loaded into the compositor, or if there is no display profile ?
>   How do you create a null transform to do an initial calibration
>   or profile ?

at startup a compositor would load the color profiles that were
configured/stored from any previous execution that it knows match the displays
it has. you mean at setup time - like when someone buys a new monitor...

i'd have the compositor use a null transform (do nothing to rgb values) UNTIL
it has calibration data for that screen. you dont have to "create" a null
transform. it's just listed in the colorspaces supported. it is the "do
nothing" colorspace.

> A simple "don't touch these pixels" flag avoids all this.

why specialize it to a flag when it actually is just an "identity transform"
really which math-wise == no nothing as a fast path which is already what
compositors do.

> > extensions can be mandatory. if clients in the greater part cease to work at
> > all without the extension present and working in a specific way.
> 
> I don't see how any of that is an immediate or even long term issue. Wayland
> applications work currently without any color management support. Adding such
> support expands the range of applications (and hence users) that can use
> Wayland.

let me roll back in time. long ago in a land far far away i was working with
x11 systems... and you then found some x11 apps that refused to work on your
xserver... because they NEEDED an 8bpp visual, but your display was just a 1
bit mono one? no emulation. apps were specifically bound to a specific depth
because thats how x11 worked. it strictly defined the output pixel value of
operations so emulation was disallowed. result - you cant run the app at all.
the not long after i had 8bpp x11 apps that refused to run on 16bpp. they also
didn't work on 1bpp. hooray! i ended up actually porting quake to 16bpp myself
(i had some ... let's say dubiously obtained source to have a linux and even
solaris/sparc (8bbp), osf1/alpha(8bpp) and linux/ix86(16bpp) port of quake to
x11...).

the problem was that you  ended up with apps that just refused to work and if
i didn't have source and the time and desire to fix them, they would have
continued to not work and if i was a regular user i would likely have just
sworn and gotten unhappy and eventually moved to a platform where this doesn't
happen.

i do not want to see this kind of thing happen again in wayland land. that's
why it matters to me. it leads to a frustrating user experience.

-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster at rasterman.com