Sub-surface protocol

[Pekka Paalanen]

Hi all,

I am currently looking into sub-surfaces, first to sketch the protocol
extension, and I have some open questions. I decided to write an
exhaustive document, so we would all be on the same page, and also to
clarify my own thoughts.


Introduction

Sub-surfaces are additional wl_surface objects that are tied to a
single application window. The "main" part of the window is the parent
wl_surface, and there can be any number of sub-surfaces.

The aim of sub-surfaces is to let the compositor do the pure compositing
work, that the application (a client) would otherwise have to do
itself. Letting the compositor do it should be more performant and
convenient.

One of the most important use cases is a video player in a window. It
has XRGB or ARGB window decorations, usually the video content in YUV,
and possibly an ARGB overlay for subtitles etc. Currently, the client
has to color-convert the video, and merge it with the decorations and
subtitles, before sending the grand ARGB buffer to the compositor. This
prohibits the use of hardware video overlays, which would offer fast and
high quality color conversions and scaling. If the YUV content was in a
buffer of its own, and the compositor would be in charge of compositing
it with the other window components, the compositor could choose to use
a hardware overlay.

The use of hardware overlays is especially important on embedded
hardware. Other possible use cases are GL widgets, and webgl and flash
in web browsers. Sub-surfaces would essentially allow different
rendering APIs to work in isolation while still efficiently producing
the pieces of a single window.


Previous Art

Jørgen Lind's extension in Qt
http://qt.gitorious.org/qt/qtwayland/blobs/master/src/extensions/sub-surface-extension.xml

Daniel Stone's experimental extension
http://cgit.collabora.com/git/user/daniels/wayland.git/commit/?h=video&id=1dc6c2307ea53699023ea0f0ce25fe62811961a3
http://cgit.collabora.com/git/user/daniels/weston.git/commit/?h=video&id=6435c41fe71e07891e71992b34169ad540d98121

Anything else?


Clipping

The term sub-surface sounds like a sub-window, which may cause one to
think, that it will be clipped to the parent surface area. I do not
think we should force or even allow such clipping.

Forcing the clipping would waste memory: for every pixel in
sub-surfaces there would have to be a pixel in the parent surface.
For instance, drawing window decorations in four sub-surfaces around
the main content would not only waste memory, but also not solve the
problem of GL applications, where one needs to reserve a margin for the
decorations. (Hello, window decoration libraries? :-)

Merely allowing clipping is just not the Wayland style. If the *client*
wants to clip something, it should not draw it in the first place.


Committing changes

Essentially we have a set of surfaces: several sub-surfaces and the one
parent wl_surface. We have the wl_surface.commit request, that
atomically updates one surface's state. This needs to be extended to
sub-surfaces, and there are several approaches.

* The pending state of all sub-surfaces is applied only on the parent
  surface commit. Sub-surface commit is a no-op.

This will nicely guarantee atomicity. The set of surfaces forms a
single application window, and all the surfaces must be updated in sync
to avoid flickering a bad composite. Resizing will absolutely require
this to work reliably and without glitches.

The downside is, that e.g. a video sink component cannot just happily
push buffers to its own sub-surface. It has to signal the application to
commit the parent surface for every single frame. Some flash, plugin,
or video APIs might not support that. Do we care?

* The sub-surface commit works like a normal surface's commit.

The video sink would be happy with this. It can just spin in its own
decoding loop and push the video into the surface, without calling back
into the application. This works fine, until something in the surface
configurations (geometry) changes. It will break horribly on resize.
Also think about flash or webgl in a browser when you scroll the page.

Therefore, we would probably want the best of the above two approaches,
which raises the following two suggestions.

* Implicit fallback to requiring parent commit on sub-surface
  configuration change. (by daniels)

This means, that sub-surface commit works like a normal surface's
commit, if the sub-surface configuration (position, buffer size) does
not change. When the configuration changes, the compositor will apply
the new sub-surface state only on the parent surface commit, and until
then it will not complete the sub-surface commit.

This will allow a video sink to run on its own, as long as the
sub-surface size or position, e.g. via wl_surface.attach, does not
change. Resizing can be handled by the application first signalling a
resize to the video sink, and then repainting and committing the parent
surface. Both the parent and the sub-surface will be updated
atomically to the screen.

For debugging, if the video sink does something unexpected, like pass
non-zero x,y to wl_surface.attach, the problem will be clearly visible
as the video will stop until the parent surface updates.

The downside is, that the client cannot force a sync to the parent
surface commit without changing the sub-surface configuration.
Therefore sub-surfaces animating in sync will be very awkward to
implement, though one can argue, that such applications should not be
using sub-surfaces to begin with.

* An explicit request toggling the behaviour of a sub-surface, whether
  it requires the parent surface commit, or commits on its own.

This gives the client explicit control on what happens on sub-surface
commit. A video player can have its video sink as autonomous, and
switch to synchronized to parent during resize. However, there might be
downsides to be discovered.

What commit behaviour should we choose?


Map and unmap

Mapping and unmapping a sub-surface follows the normal wl_surface
mapping rules, subject to the sub-surface commit behaviour, and
conditional to the parent surface: a sub-surface can be mapped only if
the parent surface is.

Would we need explicit map/unmap requests? It might depend on the
sub-surface commit behaviour with some corner cases, but I don't think
we need them.


Sub-surface configuration/geometry control

All operations about positioning and sizing sub-surfaces take place in
the parent surface's coordinate frame, and are limited to integer
pixels. No fractional pixels are allowed, and no transformations apart
from translation and scaling(?) are supported.

We probably need a wl_subsurface.set_position(x, y) request. The
initial position could be defined to be 0, 0. The position is affected
by wl_surface.attach request's x,y parameters, as usual.

The size of a normal wl_surface if defined by the wl_buffer last
attached to it. This might be insufficient for sub-surfaces, since it
does not allow using the scaling features of hardware overlays.

Should we have a request, that can set a different size for a
sub-surface, apart from the wl_buffer size?
If we did, it should probably be double-buffered state like everything
else, and follow the sub-surface commit behaviour.

What if the wl_buffer size changes without a corresponding sub-surface
size change request? Do we just scale to the set surface size
regardless of the buffer size? How would it interact with the
sub-surface commit behaviour?


Stacking order

As a client has the parent surface, and several sub-surfaces, it needs
to be able to control the z-order of all the surfaces forming the
window. Krh's proposal for the re-stacking requests was, IIUC:
- wl_subsurface.place_above(sibling wl_surface)
- wl_subsurface.place_below(sibling wl_surface)

Where the wl_surface associated with the wl_subsurface would be placed
immediately above/below of the given other wl_surface. The other
wl_surface must be either the parent or one of its sub-surfaces, but
not the one of the wl_subsurface.


Protocol sketch

My current idea is below. The wl_subsurface object is an additional
interface to the wl_surface object it was created for (not the parent).
It is similar to how wl_shell_surface is an additional interface into
wl_surface.

Destroying the wl_subsurface object does not make the wl_surface a
normal surface again, it will stay as a sub-surface until it is
destroyed.

If the parent wl_surface is destroyed, the sub-surfaces will become
inert. The only effective request for inert objects is destroy.

  <interface name="wl_subcompositor" version="1">
    <description summary="sub-surface compositing">
      The global interface exposing sub-surface compositing capabilities.
      A wl_surface, that has sub-surfaces associated, is called the
      parent surface. Sub-surfaces cannot be nested.
    </description>

    <request name="get_subsurface">
      <arg name="new_subsurface" type="new_id" interface="wl_subsurface"/>
      <arg name="surface" type="object" interface="wl_surface"/>
      <arg name="parent" type="object" interface="wl_surface"/>
    </request>
  </interface>

  <interface name="wl_subsurface" version="1">

    <enum name="commit_mode">
      <entry name="self" value="0"/>
      <entry name="parent" value="1"/>
    </enum>

    <request name="destroy" type="destructor">
    </request>

    <request name="set_position">
      <arg name="x" type="int"/>
      <arg name="y" type="int"/>
    </request>

    <request name="set_size">
      <arg name="width" type="int"/>
      <arg name="height" type="int"/>
    </request>

    <request name="place_above">
      <arg name="surface" type="object" interface="wl_surface"/>
    </request>

    <request name="place_below">
      <arg name="surface" type="object" interface="wl_surface"/>
    </request>

    <request name="set_commit_mode">
      <arg name="commit_mode" type="int"
	   summary="sub-surface commit behaviour"/>
    </request>
  </interface>

The above is written with the assumption that we will mainly handle
wl_surface objects, instead of creating a wl_something for every
wl_surface, and then tying the wl_something together. I am not sure
what the wl_something could offer, other than a lot more failure cases
to be handled. IOW, in the above, the parent surface will not have its
own wl_subsurface object.

If we ever want to allow nesting of sub-surfaces, should be enough to
just bump the wl_subcompositor version.

What should happen on wl_subsurface.get_subsurface if:
- the parent is already a sub-surface itself?
- the surface is already a sub-surface of a different parent?
- the surface is already a sub-surface of the same parent?
- the surface has already been given a role, like pointer image or
  top-level window?

Should these be fatal errors, non-fatal errors, or allowed use?
Non-fatal errors would be signalled with an additional event in
wl_subsurface, that says it is now inert due to an error.

I have not even thought about sub-surfaces' implications to input
handling or the shell yet. Sub-surfaces probably need to be able to
receive input. The shell perhaps needs a bounding box of the set of
surfaces to be able to pick an initial position for the window, etc.


Comments welcome, especially from toolkit devs who would be using this.


Thanks,
pq