[RFC v2] Wayland presentation extension (video protocol)

[Pekka Paalanen]

On Mon, 17 Feb 2014 03:23:40 +0000
"Zhang, Xiong Y" <xiong.y.zhang at intel.com> wrote:

> On Thu, 2014-01-30 at 17:35 +0200, Pekka Paalanen wrote:
> > Hi,
> > 
> > it's time for a take two on the Wayland presentation extension.
> > 
> > 
> > 		1. Introduction
> > 
> > The v1 proposal is here:
> > http://lists.freedesktop.org/archives/wayland-devel/2013-October/011496.html
> > 
> > In v2 the basic idea is the same: you can queue frames with a
> > target presentation time, and you can get accurate presentation
> > feedback. All the details are new, though. The re-design started
> > from the wish to handle resizing better, preferably without
> > clearing the buffer queue.
> > 
> > All the changed details are probably too much to describe here,
> > so it is maybe better to look at this as a new proposal. It
> > still does build on Frederic's work, and everyone who commented
> > on it. Special thanks to Axel Davy for his counter-proposal and
> > fighting with me on IRC. :-)
> > 
> > Some highlights:
> > 
> > - Accurate presentation feedback is possible also without
> >   queueing.
> > 
> > - You can queue also EGL-based rendering, and get presentation
> >   feedback if you want. Also EGL can do this internally, too, as
> >   long as EGL and the app do not try to use queueing at the same time.
> > 
> > - More detailed presentation feedback to better allow predicting
> >   future display refreshes.
> > 
> > - If wl_viewport is used, neither video resolution changes nor
> >   surface (window) size changes alone require clearing the queue.
> >   Video can continue playing even during resizes.
> Sorry, I can't understand this. Could you explain this more?
> What's the current problem for resizing window? How will you resolve it
> in presentation extensions?

Hi,

the presentation extension adds a buffer queue to wl_surfaces. The
compositor autonomously takes buffers from that queue, and applies them
to the surface when it deems the time is right. The client is notified
about this only after the fact.

Without wl_viewport, the problem with resizing is, that for the client
to do a guaranteed glitch-free resize, it has to:
1. Discard_queue, so that the compositor stops processing the queue.
2. Wait for the feedback to arrive, so that the client knows exactly
   which buffer (frame) is on screen.
3. Re-draw the frame in the new size.
4. Start queueing frames in the new size again.

As you see, that algorithm requires the client and the server to
synchronize by flushing out the whole queue and re-queue everything,
because the surface size is determined from the buffer dimensions. Step
2 is also important to avoid going backwards in the video; that is
accidentally showing a frame from an earlier time than what the
compositor already put on screen.

If the client wants to change either the surface size, or the buffer
size, (well, the same thing really, without wl_viewport), it has to do
this synchronization dance. It will have a high risk of causing a jerk
in the video playback.

Using a sub-surface with the proper sub-surface resizing algorithm, and
re-queueing in reverse chronological order would mitigate the problem,
but still have a considerable risk of producing a jerk in playback.

The solution I propose, that comes from using wl_viewport is, that a
client can resize the surface, or can resize the buffers
independently without having to discard_queue. If the client needs to
synchronize surface state updates to a particular buffer (frame), then
it still needs to do the synchronization dance.

However, video content is usually (right?) of fixed size and then gets
scaled to a right size for display. Video players would use wl_viewport
for that anyway. The added benefit explicitly enabled by the split to
surface vs. buffer state in the presentation protocol is, that you can
do surface (window) resizing without any synchronization wrt. the
buffer queue. The compositor will always scale the buffer to the right
size for the window. You do not need to send discard_queue or know what
buffer is currently presenting. And you don't need to re-draw the
buffers either when the window size changes.

OTOH, if the window size is supposed to stay the same while the video
resolution changes (for instance, QoS with a live video stream), the
client still does not need to synchronize: just keep on queueing
buffers like it always does, and the compositor takes care of the
appropriate scaling.

(When the compositor takes care of scaling with wl_viewport, it may also
be able to use overlay hardware to do superior scaling quality with
little cost.)

Therefore, I designed the presentation extension to have this, at first
perhaps rather surprising, explicit split between surface and buffer
state to specifically enable this co-operation with wl_viewport.

Note, that this extension now re-specifies what happens when a normal
wl_surface.commit is not preceded by a wl_surface.attach!

I think I need to really emphasize that on a follow-up email.


Thanks,
pq