HDR support in Wayland/Weston

Thu Jan 17 12:03:00 UTC 2019

On Wed, 16 Jan 2019 09:25:06 +0530
"Sharma, Shashank" <shashank.sharma at intel.com> wrote:

> On 1/14/2019 6:51 PM, Pekka Paalanen wrote:
> > On Thu, 10 Jan 2019 20:32:18 +0530
> > "Sharma, Shashank" <shashank.sharma at intel.com> wrote:
> >
> >> Hello All,
> >>
> >> This mail is to propose a design for enabling HDR support in
> >> Wayland/Weston stack, using display engine capabilities, and get more
> >> feedback and input from community.

*snip*

> > I understand your aim is to leverage display hardware capabilities to
> > the fullest, but we must also consider hardware that lacks some or all
> > of the conversion/mapping/other features while the monitor is well
> > HDR-capable. We also need to consider what happens when a monitor is
> > not HDR-capable or is somehow lacking. OTOH, whether a compositor
> > implements HDR support at all would be obvious in the advertised
> > Wayland globals and pixel formats.

> Very valid point. We have given good thoughts on how to handle such 
> scenarios when we are getting into, and we can come-up with some kind of 
> compositor policies which will decide if HDR video playback should be 
> allowed or not, depending on the combination of Content type, SW 
> stack(compositor and kernel), HW capabilities and connected Monitor 
> capabilities. A sample such policy may look like (also attached as txt 
> file, just in case if this table gets distorted):
> 
> +------------------------------------------------+----------------------------------+
> |Content |SW (C|K)    |HW         |Monitor       | HDR Playback 

*clip*

Talking in terms of "allowed" and "not allowed" sounds very much like
we would be needing "complicated" Wayland protocol to let applications
fail gracefully at runtime, letting them know dynamically when things
would work or not work. I believe we could do much simpler in protocol
terms as follows:

Does a compositor advertise HDR support extensions at all?

- This would depend on the compositor implementation obviously, cannot
  advertise anything without.

- Optionally, it could depend on the graphics card hardware/driver
  capabilities: if there is no card that could support HDR, then there
  is no reason advertise HDR support through Wayland, because it would
  always fall back to conversion to SDR. However, note that GPU hotplug
  might be a thing, which might bring HDR support later at runtime.

- Third, optionally again, a compositor might choose to not advertise
  HDR support if it knows it will never had a HDR-capable monitor
  attached. This is much longer stretch, and probably only for embedded
  devices you cannot plug arbitrary monitors to.

Once the Wayland HDR-related extensions have been advertised to clients
at runtime, taking them away will be hard. You may want to consider to
never revoke the extension interfaces if they have once been published
in the lifetime of a compositor instance, because revoking Wayland
globals has some caveats, mostly around clients still using HDR
extensions until they actually notice the compositor wants to redact
them. It can be made to work, but I'm not sure what the benefit would
be.

So, once a compositor advertises the extensions, they have to keep on
working at all times. Specifically this means, that if a client has
submitted a frame in HDR, and the compositor suddenly loses the ability
to physically display HDR, e.g. the only HDR monitor gets unplugged and
only SDR monitors remain, the compositor must still be able to show the
window that has HDR content lingering. So converting HDR to SDR
on-demand is a mandatory feature.

The allowed vs. not allowed is only applicable with respect to what
capabilities the compositor has advertised.

A client is not "allowed" to submit HDR content if the compositor does
not expose the Wayland extensions. Actually this is not about allowing,
but being able to submit HDR content at all: if the interfaces are not
advertised, a client simply has no interface to poke at.

Pixel formats, color spaces, and so on are more interesting. The
compositor should advertise what it supports by enumerating them
explicitly or saying what description formats it supports. Then a
client cannot use anything outside of those; if it attempts to, that
will be a fatal protocol error, not a recoverable failure.

If a client does everything according to what a compositor advertises,
it must "work": the compositor must be able to consume the client
content regardless of what will happen next, e.g. with monitor
hot-unplug, or scenegraph changing such that overlay plane is no longer
usable. This is why the fallback path through GL-renderer must exist,
and it must be able to do HDR->SDR mapping, and so on.

In summary, rather than dynamically allow or not allow, a compositor
needs to live by its promise on what works. It cannot pull the rug from
under a client by suddenly hiding the window or showing it corrupted.
That is the baseline in behaviour.

Then we can have some mechanisms in place to inform clients about
changed conditions, like sending HDR content becomes useful or stops
being useful. The first mechanism here is the wl_surface.enter/leave
events: if the wl_surface enters the first HDR output, or leaves the
last HDR output, the client will know and may adapt if it wants to.

> Also,
> - HW can always declare its capability/preference to handle HDR playback 
> using something like a DRM cap or similar.
> - We can define what to do for fallback method, if a run-time plane 
> assignment fails during atomic commit,  and so on.
> 
> IMHO, With gradual discussions in community about how and what should 
> the compositor do, i think we all can come up with how should these 
> policies look like.

*snip*

> > Since you need a proving vehicle for new kernel UABI, this may be
> > difficult.

> I know, and honestly, I was planning to tweak the compositor policies to 
> prefer GL over Overlays while plane assignment. We know that HDR 
> playback is an advance playback scenario, and we know that many HWs are 
> adding special capabilities in their graphics engine, just to handle 
> HDR, so why not use these capabilities and prefer HW overlays over GL, 
> when it comes specifically to HDR playback ? A policy which checks 
> something like this, while plane assignment:
> - If (any of the views in the list of views is HDR surface && HW is 
> capable and interested)
>      try_preparing_plane_only_state()
>   if (din't_work)
>     try_falling_for_renderer()

Yes, that is doable and good. In fact, that is what Weston's
DRM-backend does today already. It does not look at HDR status
obviously, but it does very much attempt a planes-only configuration
first, leaving the renderer unused if possible. It just that more often
than not, the planes-only mode doesn't work out and the compositor
falls back to mixed mode or renderer-only mode.

> > Maybe it would be possible to start with Wayland extensions that are
> > specific to Weston. You can cut corners there, so that you get content
> > to feed to into Weston. At some point, the Wayland extensions need to be
> > promoted to wayland-protocols and that is when you need more buy-in
> > from the community, and where the protocol design needs to be
> > well-thought. OTOH, the unstable protocols in wayland-protocols are
> > supposed to be similarly WIP, so maybe there is no need merge
> > weston-specific extensions.

> If you can elaborate this input a bit further, we can very well think 
> about this.

I just meant that if you want to experiment with Weston internals
sooner rather than later, it is ok to use weston-specific protocol
extensions if the extensions meant for wayland-protocols
standardisation are stuck in discussions. If you do that however, you
need to plan to migrate to the wayland-protocols extensions when they
have landed and drop the weston-specific extensions.

This means that you need to avoid interface name conflicts between
weston and wayland-protocols extensions.

Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20190117/b4badd02/attachment.sig>