HDR support in Wayland/Weston

[Niels Ole Salscheider]

Am Dienstag, 15. Januar 2019, 10:30:14 CET schrieb Pekka Paalanen:
> On Tue, 15 Jan 2019 13:47:07 +1100
> 
> Graeme Gill <graeme2 at argyllcms.com> wrote:
> > Pekka Paalanen wrote:
> > 
> > Hi Pekka,
> > 
> > 	thanks for your response.
> > 	
> > >> As far as I was informed, Wayland
> > >> is architected in such a way that this is not possible, since clients
> > >> have no knowledge of which display the pixels they send will end up on.
> > > 
> > > Nothing has changed there.
> > 
> > I've been pondering the various Color Management (CM) approaches to
> > working around this limitation, but I keep coming back to it
> > as the most fruitful direction to talk about. The main reasons
> > are that this implies the least extra burden on Wayland implementations,
> > and is most consonant with current application and GUI toolkit
> > CM code.
> > 
> > And in fact Wayland has already been changed in this direction already, to
> > accommodate a highly analogous requirement to that of Color Management :-
> > HiDPI. So as best I understand it, because (unlike X11) Wayland does not
> > do rendering, it is not reasonable for it to suddenly re-render
> > application graphics at hi resolution - the best that it can do is scale
> > the pixels, leading to poorer visual quality than is possible on HiDPI
> > displays. So HiDPI aware applications have to know when they are
> > rendering for a HiDPI output, and scale their buffers and rendering
> > accordingly, and tell Wayland that they have done so using
> > wl_surface.set_buffer_scale. [ I am not currently clear on how the
> > situation of a user window straddling two displays of differing DPI is
> > handled. ]
> 
> Hi Graeme,
> 
> your understanding of how HiDPI works on Wayland is correct.
> 
> If a wl_surface straddles multiple outputs simultaneously, then
> wl_surface.enter/leave events indicate the surface being on all those
> outputs at the same time. The client is expected to take all the
> entered outputs into consideration when it chooses how to render its
> image. For HiDPI, this usually means taking the maximum output scale
> from that set of outputs. The compositor will then automatically
> (down)scale for the other outputs accordingly.

> This scheme also means that the compositor does not necessarily need to
> wait for a client to render when the outputs suddenly change. It knows
> how to transform the existing image for new outputs already. The visual
> quality may jump afterwards when the client catches up, but there is no
> window blinking in and out of existence.
> 
> > The CM situation is highly analogous - like the DPI, the colorspace
> > (profile) of each output may be different for different displays, so for
> > highest quality output, something in the graphics chain needs to
> > accommodate it. To do so the relevant information is needed :- what
> > output is being rendered to, and what its characteristic is (DPI / Color
> > Profile).
> Yes, indeed.
> 
> > If done in the composer, it would need to render the graphic elements to
> > the output DPI / convert the source colorspace to the output colorspace.
> > But the composer would need the code to do rendering / convert colorspaces
> > (as well as being told what the graphic elements / source colorspace is),
> > and this is not the role Wayland has - that's the responsibility of the
> > client, so instead Wayland makes it possible for the client to know what
> > DPI it is rendering to. The analogous facility for CM is for the client
> > to know what output colorspace it is rendering for.
> 
> Yes and no. Yes, we do and should let clients know what kind of outputs
> their contents will be shown on. However, we will in any case need the
> compositor to be able to do the full and correct conversion from what
> ever the client has submitted to what is correct for a specific output,
> because nothing guarantees that those two would always match.
> 
> One wl_surface on multiple outputs is an obvious case where one buffer
> rendered by a client cannot match all the outputs it is shown on. The
> other case is transitions between outputs, where we cannot have the
> compositor wait for the client to re-draw with new color parameters.

I think the last proposal of a color management protocol that we discussed 
does that. It contains the device link profiles and it also allows the client 
to query the profile of wl_outputs. With that, an application can display 
accurate colors in nearly every situation, even on multiple screens 
simultaneously. But still the compositor can do it's best to provide a good 
output in some corner cases (e. g. when a new screen is activated and the 
application has not rendered a new frame yet). Once the application reacts to 
that change the output will be perfect again.

> > The clean/simplest approach to HDR is to treat it as just another output
> > colorspace, where it is up to the application to render the color
> > it intends to display, and the Wayland compositor is compositing
> > everything
> > for that output in that HDR colorspace.
> 
> Agreed.
> 
> > Now practically speaking this would assume that all Wayland clients
> > connected to an HDR display in HDR mode are CM/HDR aware, which is rather
> > unlikely. So some backward compatibility modes might be highly desirable
> > (I have some thoughts on that), but in any case, it would also help the
> > quality of such backward compatibility _and_ compositing (i.e. linear
> > light compositing option), if Wayland at least had access to the output
> > color profiles. So there is a lot of advantage in Wayland providing the
> > registry/API of output color profiles both for itself, and clients.
> 
> That backward compatibility / fallback is an integral part of the image
> transformations the compositor must be able to do in any case. So yes,
> I agree.
> 
> For instance, the whole HDR support project could start with:
> - letting Weston recognize HDR-capable monitors and fetch their info
> - turn on HDR video mode
> - patch GL-renderer to convert all old (sRGB or whatever default when
>   nothing is said about color space and other details) content to be
>   suitable for HDR display
> This makes no use of the monitor HDR capability, but it would start
> with the most important feature: backward compatibility. Adding the
> Wayland protocol to allow clients to reasonably produce and submit HDR
> content could come after this as a next phase.
> 
> What details we actually need to expose via Wayland about the outputs
> or compositor blending spaces or something else I will gladly leave to
> those who actually know about color management.
> 
> > > Wayland and apps need to provide the compositor all the necessary
> > > information for the compositor to do all the conversions, mapping and
> > > blending correctly, if it has to.
> > 
> > And perhaps it shouldn't have to.
> 
> I'm not sure I understand what you mean. Above you seemed to agree with
> this. Maybe I should emphasise the "if it has to"? That is, when for
> whatever reason, the client provided content is not directly suitable
> for the output it will be displayed on.
> 
> > > This is because an application will provide only one image for a
> > > wl_surface, and the compositor may show that on any number of any kind
> > > of outputs at once.
> > 
> > That's a big problem. That assumes either that all displays are
> > interchangeable at the pixel level (they are not - they have different
> > DPI and Colorspaces/Gamut/HDR capability), or that Wayland has to know
> > how to re-render to accommodate those difference (and Wayland doesn't do
> > rendering, and may not want to include Colorspace conversion machinery or
> > HDR colorspace/conversion machinery).
> It will be the compositor that will automatically convert images to
> whatever output they will be displayed on, just like the automatic
> scaling with HiDPI. Wayland only needs to relay enough information that
> the compositor can do that when necessary.
> 
> > > Nothing prevents adding more protocol to give apps more hints to behave
> > > more optimally wrt. to the compositor's internal pipeline.
> > 
> > There's a difference between hints that help speed and quality but
> > still burden the compositor with doing display dependent conversions,
> > and an approach that removes hat burden completely to the client when
> > it does the rendering.
> 
> That burden cannot be eliminated in a general desktop environment.
> There will always be situations where the compositor has to convert,
> because a client has not or could not provide directly suitable content.
> 
> If we were to limit ourselves to narrow use cases outside of a generic
> desktop system, then eliminating that burden might be possible. For
> devices like TVs that would be highly desirable even. In the closed
> system of a TV, all apps and all outputs are known beforehand, so the
> operating system can simply avoid the general anything-to-anything
> situations. Desktop systems are not that limited.
> 
> Weston needs to have all those correct conversions implemented. If
> Weston gets used in an environment where those conversions are not
> necessary, then Weston will simply not execute them.
> 
> > > Correct, and Shashank is not proposing anything different. The
> > > per-channel lookup curves are a compositor internal detail, always
> > > programmed by the compositor correctly according to what it happens to
> > > be showing each monitor refresh on that specific monitor.
> > 
> > Since the compositor is not Color Management aware, then by definition
> 
> The compositor must be made Color Management aware at some level.
> Otherwise it will not expose any of the Wayland extensions that would
> allow clients to submit content aside from the undefined(?) default
> color space, or the extensions that would describe the output color
> properties.
> 
> > it can't set the CRTC to the correct values ("correct" in terms
> > of the color sensitive end users intentions for how they need their
> > systems
> > and applications to work.) Now if the compositor was CM aware, it could
> > choose whether to implement CM display calibration curves by using
> > the hardware CRTC, or by implementing it in some other fashion such
> > as with a shader, or (ideally) by downloading the curve to the display
> > itself (many high end displays have this capability, and the
> > commercial CM tools do exactly that. How that could work with
> > Wayland is another problem to ponder.)
> 
> Yes, a compositor must implement all that, but this is now slipping to
> the topic of calibration, which is very much off-scope for today. We
> only want to consider how applications produce and provide content with
> specific color properties for now.
> 
> Let's concentrate on getting proper content from clients today, and
> leave the matter of analysis and calibration for another time. It is
> already a big topic as is.
> 
> 
> Thanks,
> pq


-- 
Niels Ole Salscheider
Wissenschaftlicher Mitarbeiter
ISPE MPS

FZI Forschungszentrum Informatik
Haid-und-Neu-Str. 10–14
76131 Karlsruhe, Germany
Tel.: +49 721 9654-264
salscheider at fzi.de

FZI Forschungszentrum Informatik
Stiftung des bürgerlichen Rechts
Stiftung Az: 14-0563.1 Regierungspräsidium Karlsruhe
Vorstand: Prof. Dr. Andreas Oberweis, Jan Wiesenberger,
Prof. Dr.-Ing. J. Marius Zöllner
Vorsitzender des Kuratoriums: Ministerialdirigent Günther Leßnerkraus