[RFC] Sub-surface protocol and implementation v1

[John Kåre Alsaker]

My goals for a subsurface implementation are these:
- Allow nesting to ease interoperability for client side code.
- Allow a surface without any content to have an input region and let
the content be presented in a number of adjacent subsurfaces. This
would simplify input handling by a lot.
- Allow clients to commit a set of surfaces.

On Tue, Jan 8, 2013 at 8:50 AM, Pekka Paalanen <ppaalanen at gmail.com> wrote:
> Hi John,
>
> thanks for the comments, I have some more questions about your
> suggestions.
>
>
> On Mon, 7 Jan 2013 16:56:47 +0100
> John Kåre Alsaker <john.kare.alsaker at gmail.com> wrote:
>
>> On Fri, Dec 21, 2012 at 12:56 PM, Pekka Paalanen <ppaalanen at gmail.com> wrote:
>> > Hi all,
>> >
>> > we started a discussion about sub-surfaces in Wayland in [1].
> ...
>> >
>> > There are still many open questions, ranging from implementation
>> > details I noted in code comments and commit messages, to heavy protocol
>> > additions like clipping. Some of the open questions are:
>> > - how should commits work in parent vs. sub-surfaces?
>> Commit should work the same way. It should commit itself and all it's
>> children. Furthermore it should commit all reordering of it's
>> immediate children.
>
> Could you give some rationale why this is preferred to any other way,
> for instance sub-surface commit needing a main surface commit to apply
> all the pending state?
We don't want to keep another copy of the surface state around and
using dummy surfaces and nesting we can commit a set of surfaces as we
please.

>
> Collecting the reasons why some method should be preferred is
> important, even if they are somehow obvious, so we can choose.
>
>> > - do we allow nested sub-surfaces?
>> Yes, this makes it easier for client code to cooperate. If a library
>> wants to use subsurfaces it shouldn't need any special handling with
>> the application to achieve that. It also allows clients to commit a
>> group of sub-surfaces by using a dummy parent.
>
> Do you mean a library creates a dummy sub-surface as a parent for other
> sub-surfaces it creates, so it can commit then all in one go?
Yes, although it may also use a real sub-surface and set the others as
children if that's convenient.

>
> How would we implement the dummy parent? There is no concept of a dummy
> surface in the protocol yet, and currently mapping a sub-surface
> depends on mapping the immediate parent.
A dummy parent would simply be a surface without content. It would be
mapped (by the shell, it should be left out when rendering). It would
have the size of it's children, or we could add a way to specify the
size of a surface without buffers, which could be shared with a
scaling implementation. I'm not very clear on what sizes are used for
though.

>
> I believe breaking this mapping requirement chain would require a lot
> more complex rules on what is mapped and when. OTOH, creating a dummy
> surface with e.g. attaching a 1-pixel 100% transparent wl_buffer is in
> my opinion a hack to be reserved for cases where we simply cannot change
> the protocol anymore. We are now designing the protocol for the first
> time, so let's try to avoid such workarounds to begin with.
I don't see any reasons for why dummy surface would add complexity.
Also using a 1-pixel transparent buffer would result in horrible
performance at the moment and would require odd optimizations.

>
> Do you have a use case for this?
Dummy surface are a useful way to group surfaces. Without them, when
deleting the first surface in a group (which would be the parent
instead of the dummy surface), you'd have to delete all the surfaces
in the group and recreate all but the first. With re-parenting this is
less crazy, but still requires quite a few operations.

>
> As for nesting in itself, it would be simple to just allow it, I think,
> and I have not found any good reason to forbid it yet.
>
>> > - do we want a completely orthogonal interface for clipping and scaling,
>> >   or should it be limited to sub-surfaces?
>> Clipping and/or scaling should work for all kinds of surfaces. I don't
>> see any reason to limit it to sub-surfaces.
>
> Yeah, I have a feeling it would make sense as an orthogonal interface.
> It will be a huge change in any case, since surface size cannot be
> determined by a wl_buffer alone anymore.
Surface size not being determinable by wl_buffer's size is also a
requirement of dummy surfaces with a input region.

>
> Regarding sub-surfaces, there is the question of whether clipping
> should be controlled within the sub-surface update-commit cycle or
> decoupled from it, for instance a video overlay driven by an autonomous
> decoding library where the application needs to clip it; does it need
> to communicate clipping to the library, set clipping on its own, or
> always commit every video frame. Positioning the sub-surface wrt. to
> the window is up for the application, so clipping might be as well.
> Hence clipping may have some interactions with sub-surfaces, and I
> don't know how any existing software does it.
I think existing software use window systems with trees where the
children. With the ability to set the size of a surface, we could add
a flag for that. We could also add explicit sizes for clipping of
children. I think that's the best approach to support that use case.
We always want regular clipping to be on the surface commit cycle so
it can be used to view a part of a wl_buffer.

>
>> > - should we forbid reparenting?
>> > - should sub-surface role be revertible, instead of permanent?
>> I see this as a nice source for race conditions, although it may turn
>> out to be useful.
>
> Maybe I should follow the other existing cases, that is cursor
> surfaces, and maybe d&d had a role in use, too, for the drag icon.
>
>> > - should a sub-surface be restricted to its parent's area?
>> It should not, especially since we don't have a good way to define the
>> parent area. Dummy surfaces whose only purpose is to hold sub-surfaces
>> depend on them not being restricted to the non-existing parent area.
>
> Right now I agree, parent area is indeed a problematic concept in the
> core protocol, since we have uses for completely transparent areas.
> Another reason is to be able to compose a window from adjacent,
> non-overlapping, surfaces.
>
>> > [1] http://lists.freedesktop.org/archives/wayland-devel/2012-December/006623.html
>> > [2] http://lists.freedesktop.org/archives/wayland-devel/2012-December/006837.html
>> > [3] http://people.collabora.com/~pq/subsurface-1.png
>
>> Sub-surfaces should also be able to get keyboard focus, no special
>> case is required for that. We should also make sure that dummy
>> surfaces without any attached buffers, but only sub-surfaces work
>> correctly with or without a input region.
>
> Yes, I am fixing the keyboard focus thing right now. I did the server
> part, but now fixing toytoolkit to work right with it is a heavy task.
>
> Sorry? I didn't quite get what you meant by the latter sentence.
I mean that we should make sure a surface without a buffer works as a
parent for subsurfaces both when it has a input region and when it
doesn't.

>
> Oh, and about using dummy surfaces: mapping is a concept related to the
> shell. It is left for the shell to define when mapping happens. I think
> this might be a bit unclearly specified currently. I'm not sure "dummy"
> could be synonymous to "not mapped" surface, and interacting between
> core and shell protocol like this to excercise a core protocol feature
> (committing simultaneously several sub-surfaces) feels messy to me.
>
Dummy surfaces should get mapped normally (if it's assigned a size).
The shell only cares about the group of surfaces.

> Hm, maybe instead of a dummy surface, a library could just use the
> given wl_surface as the parent for any other sub-surfaces it might want
> to create. Would this be enough?
I don't see how this would be different from dummy surfaces
(wl_surfaces without a wl_buffer), unless they actually have a
wl_buffer, in which case I gave an example of an operation which isn't
fun.

>
>
> Thanks,
> pq