HDR support in Wayland/Weston
Sharma, Shashank
shashank.sharma at intel.com
Wed Jan 16 03:55:06 UTC 2019
Hello Pekka,
Thanks a lot for your comments, and inputs on design, stability and
general accessibility across all platforms.
Please find my comments, inline.
Regards
Shashank
On 1/14/2019 6:51 PM, Pekka Paalanen wrote:
> On Thu, 10 Jan 2019 20:32:18 +0530
> "Sharma, Shashank" <shashank.sharma at intel.com> wrote:
>
>> Hello All,
>>
>> This mail is to propose a design for enabling HDR support in
>> Wayland/Weston stack, using display engine capabilities, and get more
>> feedback and input from community.
>> Here are few points (you might already know these), about HDR
>> framebuffers, videos and displays:
>> - HDR content/buffers are composed in REC2020 colorspace, with bit depth
>> 10/12/16 BPC. Some of the popular formats are P010,P012,P016.
>> - HDR content come with their own Metadata to be applied to get the
>> right luminance at the display device.
>> - The metadata can be of two type 1. static 2. dynamic . For
>> simplicity, this solution is focusing on static HDR only (HDR10 standard)
>> - HDR content also provide its supported EOTF (electro optical transfer
>> function) information, which is a curve (like SRGB gamma curve). One
>> popular EOTF is PQ(ST2084).
>> - HDR capable displays mention their EOTF and HDR metadata support
>> information in EDID CEA-861-G blocks.
>> - Normal SRGB buffers are composed in SRGB color space following REC709
>> specifications.
>> - For accurate blending in display engines, we need to make sure following:
>> - All the buffers are in same colorspace (Rec 709 or Rec 2020)
>> - All the buffers are liner (gamma/EOTF removed)
>> - All the buffers are tone mapped in same zone (HDR or SDR)
>>
>> Please refer to the block diagram below, which presents a simple case of
>> a HDR P010 movie playback, with HDR buffers as video buffers, and SDR
>> buffers as subtitles. The subsystem looks and works like this:
>> - A client decodes the buffer (using FFMpeg for example) and gets the
>> two buffers, one with video (HDR) and one subtitles (SDR)
>> - Client passes following information to the compositor:
>> - The actual buffers
>> - Their colorspace infromation, BT2020 for HDR buffer, REC709 for
>> SDR buffer (planning to add a new protocol extension for this)
>> - The HDR metadata of the content (planning to add new protocol
>> for this)
>>
>> - Compositors actions:
>> - Reads the End display's HDR capabilities from display EDID. Assume
>> its an HDR HDMI monitor.
>> - Compositor tone maps every view's framebuffer to match tone of end
>> display, applying a libVA filter. In this example:
>> - The SDR subtitles frame will go through SDR to HDR tone
>> mapping (called S2H)
>> - The HDR video frame will go through HDR to HDR tone mapping
>> (called H2H) if the HDR capabilities of monitor and content are different.
>> - Now both the buffers and the monitor are in the same tone
>> mapped range.
>> - As the end display is HDR capable, and one of the content frame
>> is HDR, the compositor will prepare all other planes for color space
>> conversion (CSC) from REC709->REC2020 using plane CSC property.
>> - As the CSC and blending should be done in liner space, compositor
>> will also use plane level degamma to make the buffers linear.
>> - These actions will make sure that, during blending:
>> - All the buffers are in same colorspace (REC2020)
>> - All the buffers are linear
>> - All the buffers are tone mapped (HDR)
>> - The plane level color properties patch, for DRM can be found
>> here: https://patchwork.freedesktop.org/series/30875/
>> - Now, in order to re-apply the HDR curve, compositor will apply
>> CRTC level gamma, so that the output buffer is non-linear again.
>> - To pass the output HDR information to kernel, so that it can
>> create and send AVI-info-frames to HDMI, compositor will set Connector
>> HDR metadata property.
>> - Code for the same can be found here:
>> https://patchwork.freedesktop.org/series/25091/
>> - And they will ever live happily after :).
>>
>> Please provide inputs, feedbacks and suggestions for this design and
>> plan, so that we can improve out half cooked solution, and start sending
>> the patches.
> Hi Shashank,
>
> this is a major feature that would be awesome to have in Weston, but
> it's also a big effort to design, implement, and maintain. To ease the
> maintenance, we will need some serious work on the test suite, which
> currently cannot even run the GL-renderer.
I agree, Weston's stability and maintenance must be high priority.
That's why I was thinking that instead of enabling such a huge feature
in a single shot, and trying to cover all possible cases and
combinations, we can split it into small modular subset of features, in
form of small patch series, review it well, and add it slowly in the
mainstream with proper precaution.
These subsets could be ( just example)
- enabling single HDR view (like a movie playback), in fullscreen mode,
targeting only one target colorspace (say BT2020) and one type of HDR
metadata (say Static HDR metadata, PQ ST 2084 EOTF, HDR 10 type)
If we find this subset stable, working without any regressions and
stable enough, we can move to phase-2
- enabling HDR view in combination with one SDR view (as shown in the
design, movie playback with subtitles), again in fullscreen mode, for
one target colorspace, and one type of HDR metadata handling.
if this works well again, we can move to phase -3, which could be:
- enabling HDR view playnack in full windowed mode, with multiple SDR
views, and one HDR view, for one target colorspace and one type of HDR
metadata.
And so on.
In this way, in a slow and granular process, we would be able to enable
the mega feature in slow, careful and well tested steps.
> I understand your aim is to leverage display hardware capabilities to
> the fullest, but we must also consider hardware that lacks some or all
> of the conversion/mapping/other features while the monitor is well
> HDR-capable. We also need to consider what happens when a monitor is
> not HDR-capable or is somehow lacking. OTOH, whether a compositor
> implements HDR support at all would be obvious in the advertised
> Wayland globals and pixel formats.
Very valid point. We have given good thoughts on how to handle such
scenarios when we are getting into, and we can come-up with some kind of
compositor policies which will decide if HDR video playback should be
allowed or not, depending on the combination of Content type, SW
stack(compositor and kernel), HW capabilities and connected Monitor
capabilities. A sample such policy may look like (also attached as txt
file, just in case if this table gets distorted):
+------------------------------------------------+----------------------------------+
|Content |SW (C|K) |HW |Monitor | HDR Playback
? |
+-----------------------------------------------------------------------------------+
| | | | HDR |
Allowed |
| | | HDR
+-------------------------------------------------+
| | HDR | | SDR | Allowed(match
monitor with H2S)|
| | +-------------------------------------------------------------+
| | | | HDR | Not
allowed |
| | | SDR
+-------------------------------------------------+
| HDR | | | SDR | Now
allowed |
|
+--------------------------------------------------------------------------+
| | | | HDR | Not
allowed |
| | | HDR
+--------------------------------------------------+
| | SDR | | SDR | Not
allowed |
| | +-------------------------------------------------------------+
| | | | HDR | Now
allowed |
| | | SDR | SDR
+------------------------------------------------+----------------------------------+
| | | | HDR | Allowed(match
monitor S2H) |
| | | HDR | SDR | Allowed(nothing
required) |
| | +-------------------------------------------------------------+
| | HDR | | HDR | And so
on |
| | | SDR | SDR
| |
| SDR | | | | |
|
+--------------------------------------------------------------------------+
| | | | HDR | And so
on |
| | | HDR | SDR
| |
| | SDR
+-------------------------------------------------------------+
| | | | HDR | And so
on |
| | | SDR | SDR
| |
+--------+------------+-----------+--------------+----------------------------------+
Also,
- HW can always declare its capability/preference to handle HDR playback
using something like a DRM cap or similar.
- We can define what to do for fallback method, if a run-time plane
assignment fails during atomic commit, and so on.
IMHO, With gradual discussions in community about how and what should
the compositor do, i think we all can come up with how should these
policies look like.
>
> Do we want to support HDR output in some way even if the display engine
> (in the graphics card) does not support all the features you want to
> use?
The playback compositor policy can decide this case also (HW section of
the table above)
>
> That question is on some parts already answered by whether we can
> always rely on display engine hardware features: no, we cannot. The
> reason is that Weston cannot generally guarantee that it can use a
> hardware plane for a specific wl_surface. Hardware planes are used
> opportunistically, and we must always have an equivalent fallback path
> which usually means the GL-renderer. Users should not be able to see a
> difference between a GL composited and a display engine composited
> scene.
I agree, this is very reasonable to have GL path as fallback, always.
But also, this means, for every feature, Weston's implementation will be
blocked until we have a parallel GL path to handle stuff. It seems very
safe, but IMHO, in compositor, we should be ready to tweak into our
priorities if we already know that we are running on a very capable HW,
which declares its strengths and inclinations in form of a DRM cap or
something similar.
Honestly, this is something which touches the stability, and I would
always respect the opinion of a maintainer who knows this stuff better
than me :-).
> Needing the fallback from hardware planes determines the minimum of
> what GL-renderer needs to implement. Whether there is anything more is
> an open question. Using hardware planes in the test suite is difficult,
> so having a renderer implementation will help maintenance too.
I agree.
> What if we have only SDR monitors? Or a mixed set of SDR and HDR
> monitors?
This situation can also be covered in the playback decision taken above.
A good thing if we are going to support static HDR as stage-1 is, you
have to take the decision only once, when we are starting the playback,
and that would be valid for the life of the content/playback session. Of
course it gets complicated when we would plan to support Dynamic HDR
support (like HDR 10+), where the metadata can change per frame.
> I suppose it would be good for apps to know if HDR content is
> preferable or not. You could hook up to the wl_output and
> wl_surface.enter/leave mechanism that is already used to control e.g.
> HiDPI preference (buffer/output scale). It means clients know which
> outputs a wl_surface is present on, and somehow choose how to draw the
> content. The compositor then automatically converts, as necessary, to
> any outputs that do not exactly match the app provided content.
I agree, in fact, we can create an interface from DRM cap to apps, which
will let the applications know if this SW+HW+Monitor combination prefers
to support HDR playback.
> If switching the content type is too costly for apps to do during
> window moves, then there could be a global recommendation as well,
> either with explicit protocol extension or intended to be inferred from
> the properties of all the wl_outputs exposed by the compositor.
Seems like a good suggestion too.
> **
>
> What we need for this is a plan that can be implemented and merged in
> small steps, while never knowingly regressing anything.
Exactly.
> I've been talking about GL-renderer, because I don't think implementing
> any of this with Pixman-renderer would be useful. I think you can just
> forget about Pixman-renderer aside from honouring renderer capabilities
> when advertising support to Wayland clients.
Sure. Agree.
>
> Before we can use hardware plane capabilities, we probably need the
> fallback path, GL-renderer, to be up to par. I hope the HDR support can
> be split into smaller bits somehow, where we could iterate between
> extending GL-renderer and taking more hardware plane features into use,
> if that makes sense.
Sure, let me work on that part.
> Since you need a proving vehicle for new kernel UABI, this may be
> difficult.
I know, and honestly, I was planning to tweak the compositor policies to
prefer GL over Overlays while plane assignment. We know that HDR
playback is an advance playback scenario, and we know that many HWs are
adding special capabilities in their graphics engine, just to handle
HDR, so why not use these capabilities and prefer HW overlays over GL,
when it comes specifically to HDR playback ? A policy which checks
something like this, while plane assignment:
- If (any of the views in the list of views is HDR surface && HW is
capable and interested)
try_preparing_plane_only_state()
if (din't_work)
try_falling_for_renderer()
> Maybe it would be possible to start with Wayland extensions that are
> specific to Weston. You can cut corners there, so that you get content
> to feed to into Weston. At some point, the Wayland extensions need to be
> promoted to wayland-protocols and that is when you need more buy-in
> from the community, and where the protocol design needs to be
> well-thought. OTOH, the unstable protocols in wayland-protocols are
> supposed to be similarly WIP, so maybe there is no need merge
> weston-specific extensions.
If you can elaborate this input a bit further, we can very well think
about this.
> Personal experience says that with something this big, you end up
> needing to restructure and clean up Weston's current code before you
> get to the main matter. The earlier we get any patch merged, the less
> work you need carrying them forward until the kernel bits have landed.
Agree.
> Will be interesting to see how this unravels.
Thanks for the inputs.
- Shashank
>
> Thanks,
> pq
-------------- next part --------------
+------------------------------------------------+----------------------------------+
|Content |SW (C|K) |HW |Monitor | HDR Playback ? |
+-----------------------------------------------------------------------------------+
| | | | HDR | Allowed |
| | | HDR +-------------------------------------------------+
| | HDR | | SDR | Allowed(match monitor with H2S)|
| | +-------------------------------------------------------------+
| | | | HDR | Not allowed |
| | | SDR +-------------------------------------------------+
| HDR | | | SDR | Now allowed |
| +--------------------------------------------------------------------------+
| | | | HDR | Not allowed |
| | | HDR +--------------------------------------------------+
| | SDR | | SDR | Not allowed |
| | +-------------------------------------------------------------+
| | | | HDR | Now allowed |
| | | SDR | SDR
+------------------------------------------------+----------------------------------+
| | | | HDR | Allowed(match monitor S2H) |
| | | HDR | SDR | Allowed(nothing required) |
| | +-------------------------------------------------------------+
| | HDR | | HDR | And so on |
| | | SDR | SDR | |
| SDR | | | | |
| +--------------------------------------------------------------------------+
| | | | HDR | And so on |
| | | HDR | SDR | |
| | SDR +-------------------------------------------------------------+
| | | | HDR | And so on |
| | | SDR | SDR | |
+--------+------------+-----------+--------------+----------------------------------+
More information about the wayland-devel
mailing list