HDR support in Wayland/Weston

[Pekka Paalanen]

On Tue, 15 Jan 2019 13:47:07 +1100
Graeme Gill <graeme2 at argyllcms.com> wrote:

> Pekka Paalanen wrote:
> 
> Hi Pekka,
> 	thanks for your response.
> 
> >> As far as I was informed, Wayland
> >> is architected in such a way that this is not possible, since clients
> >> have no knowledge of which display the pixels they send will end up on.  
> > 
> > Nothing has changed there.  
> 
> I've been pondering the various Color Management (CM) approaches to
> working around this limitation, but I keep coming back to it
> as the most fruitful direction to talk about. The main reasons
> are that this implies the least extra burden on Wayland implementations,
> and is most consonant with current application and GUI toolkit
> CM code.
> 
> And in fact Wayland has already been changed in this direction already, to
> accommodate a highly analogous requirement to that of Color Management :- HiDPI.
> So as best I understand it, because (unlike X11) Wayland does not do
> rendering, it is not reasonable for it to suddenly re-render application
> graphics at hi resolution - the best that it can do is scale the pixels,
> leading to poorer visual quality than is possible on HiDPI displays.
> So HiDPI aware applications have to know when they are rendering for a HiDPI
> output, and scale their buffers and rendering accordingly, and tell Wayland that
> they have done so using wl_surface.set_buffer_scale. [ I am not currently
> clear on how the situation of a user window straddling two displays of
> differing DPI is handled. ]

Hi Graeme,

your understanding of how HiDPI works on Wayland is correct.

If a wl_surface straddles multiple outputs simultaneously, then
wl_surface.enter/leave events indicate the surface being on all those
outputs at the same time. The client is expected to take all the
entered outputs into consideration when it chooses how to render its
image. For HiDPI, this usually means taking the maximum output scale
from that set of outputs. The compositor will then automatically
(down)scale for the other outputs accordingly.

This scheme also means that the compositor does not necessarily need to
wait for a client to render when the outputs suddenly change. It knows
how to transform the existing image for new outputs already. The visual
quality may jump afterwards when the client catches up, but there is no
window blinking in and out of existence.

> The CM situation is highly analogous - like the DPI, the colorspace (profile)
> of each output may be different for different displays, so for highest
> quality output, something in the graphics chain needs to accommodate it.
> To do so the relevant information is needed :- what output is being
> rendered to, and what its characteristic is (DPI / Color Profile).

Yes, indeed.

> If done in the composer, it would need to render the graphic elements to
> the output DPI / convert the source colorspace to the output colorspace.
> But the composer would need the code to do rendering / convert colorspaces
> (as well as being told what the graphic elements / source colorspace is),
> and this is not the role Wayland has - that's the responsibility of the
> client, so instead Wayland makes it possible for the client to know what DPI
> it is rendering to. The analogous facility for CM is for the client to know
> what output colorspace it is rendering for.

Yes and no. Yes, we do and should let clients know what kind of outputs
their contents will be shown on. However, we will in any case need the
compositor to be able to do the full and correct conversion from what
ever the client has submitted to what is correct for a specific output,
because nothing guarantees that those two would always match.

One wl_surface on multiple outputs is an obvious case where one buffer
rendered by a client cannot match all the outputs it is shown on. The
other case is transitions between outputs, where we cannot have the
compositor wait for the client to re-draw with new color parameters.

> The clean/simplest approach to HDR is to treat it as just another output
> colorspace, where it is up to the application to render the color
> it intends to display, and the Wayland compositor is compositing everything
> for that output in that HDR colorspace.

Agreed.

> Now practically speaking this would assume that all Wayland clients connected
> to an HDR display in HDR mode are CM/HDR aware, which is rather unlikely. So
> some backward compatibility modes might be highly desirable (I have
> some thoughts on that), but in any case, it would also help the quality
> of such backward compatibility _and_ compositing (i.e. linear light
> compositing option), if Wayland at least had access to the output color
> profiles. So there is a lot of advantage in Wayland providing the
> registry/API of output color profiles both for itself, and clients.

That backward compatibility / fallback is an integral part of the image
transformations the compositor must be able to do in any case. So yes,
I agree.

For instance, the whole HDR support project could start with:
- letting Weston recognize HDR-capable monitors and fetch their info
- turn on HDR video mode
- patch GL-renderer to convert all old (sRGB or whatever default when
  nothing is said about color space and other details) content to be
  suitable for HDR display
This makes no use of the monitor HDR capability, but it would start
with the most important feature: backward compatibility. Adding the
Wayland protocol to allow clients to reasonably produce and submit HDR
content could come after this as a next phase.

What details we actually need to expose via Wayland about the outputs
or compositor blending spaces or something else I will gladly leave to
those who actually know about color management.

> > Wayland and apps need to provide the compositor all the necessary
> > information for the compositor to do all the conversions, mapping and
> > blending correctly, if it has to.  
> 
> And perhaps it shouldn't have to.

I'm not sure I understand what you mean. Above you seemed to agree with
this. Maybe I should emphasise the "if it has to"? That is, when for
whatever reason, the client provided content is not directly suitable
for the output it will be displayed on.

> 
> > This is because an application will provide only one image for a
> > wl_surface, and the compositor may show that on any number of any kind
> > of outputs at once.  
> 
> That's a big problem. That assumes either that all displays are interchangeable
> at the pixel level (they are not - they have different DPI and Colorspaces/Gamut/HDR
> capability), or that Wayland has to know how to re-render to accommodate those
> difference (and Wayland doesn't do rendering, and may not want to include
> Colorspace conversion machinery or HDR colorspace/conversion machinery).

It will be the compositor that will automatically convert images to
whatever output they will be displayed on, just like the automatic
scaling with HiDPI. Wayland only needs to relay enough information that
the compositor can do that when necessary.

> > Nothing prevents adding more protocol to give apps more hints to behave
> > more optimally wrt. to the compositor's internal pipeline.  
> 
> There's a difference between hints that help speed and quality but
> still burden the compositor with doing display dependent conversions,
> and an approach that removes hat burden completely to the client when
> it does the rendering.

That burden cannot be eliminated in a general desktop environment.
There will always be situations where the compositor has to convert,
because a client has not or could not provide directly suitable content.

If we were to limit ourselves to narrow use cases outside of a generic
desktop system, then eliminating that burden might be possible. For
devices like TVs that would be highly desirable even. In the closed
system of a TV, all apps and all outputs are known beforehand, so the
operating system can simply avoid the general anything-to-anything
situations. Desktop systems are not that limited.

Weston needs to have all those correct conversions implemented. If
Weston gets used in an environment where those conversions are not
necessary, then Weston will simply not execute them.

> > Correct, and Shashank is not proposing anything different. The
> > per-channel lookup curves are a compositor internal detail, always
> > programmed by the compositor correctly according to what it happens to
> > be showing each monitor refresh on that specific monitor.  
> 
> Since the compositor is not Color Management aware, then by definition

The compositor must be made Color Management aware at some level.
Otherwise it will not expose any of the Wayland extensions that would
allow clients to submit content aside from the undefined(?) default
color space, or the extensions that would describe the output color
properties.

> it can't set the CRTC to the correct values ("correct" in terms
> of the color sensitive end users intentions for how they need their systems
> and applications to work.) Now if the compositor was CM aware, it could
> choose whether to implement CM display calibration curves by using
> the hardware CRTC, or by implementing it in some other fashion such
> as with a shader, or (ideally) by downloading the curve to the display
> itself (many high end displays have this capability, and the
> commercial CM tools do exactly that. How that could work with
> Wayland is another problem to ponder.)

Yes, a compositor must implement all that, but this is now slipping to
the topic of calibration, which is very much off-scope for today. We
only want to consider how applications produce and provide content with
specific color properties for now.

Let's concentrate on getting proper content from clients today, and
leave the matter of analysis and calibration for another time. It is
already a big topic as is.


Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20190115/615dc052/attachment.sig>