Nesting sub-surfaces?

[David Herrmann]

Hi Pekka

On Wed, Mar 6, 2013 at 1:49 PM, Pekka Paalanen <ppaalanen at gmail.com> wrote:
> Hi,
>
> I have been pondering about sub-surface nesting, i.e. allowing
> sub-sub-surfaces and so on, and I have hard time figuring out how
> commits should work.
>
> The v2 sub-surface protocol proposal is here:
> http://cgit.collabora.com/git/user/pq/weston.git/tree/protocol/subsurface.xml?h=subsurface-v2
>
> Adding support for nesting does not require any API changes to the
> protocol. We just need to define, that it is ok to have a sub-surface
> as a parent for another sub-surface, and how commits work. Commit
> behaviour is again the hard part.
>
> Suppose we have a window consisting of the following:
>
> main surface, wl_surface A
>         ↳ sub-surface, wl_surface B
>
> Every surface has current state (how it is currently drawn on screen),
> and pending state (to accumulate pieces of state, that will then be
> made current atomically on wl_surface.commit).
>
> Additionally, sub-surfaces have a state cache. When a sub-surface
> itself is explicitly committed, the pending state either becomes
> current, or goes to the cache, depending on the commit mode of the
> sub-surface. The pending state can only be used when a commit request
> was received for the surface itself, never via parent commit.
>
> If the commit mode for the sub-surface B is parent-cached, then
> B.commit() will push the pending state into the cache. The cached state
> is made current on A.commit(), so that all surfaces get updated
> atomically.
>
> If the commit mode for the sub-surface B is independent, then
> B.commit() will make the pending state current. A.commit() does not
> affect B's current state, except for the wl_subsurface.set_position.
>
> The position set by a wl_subsurface.set_position request is always
> stored, and applied only on parent surface commit (A.commit()). This
> way the contents of wl_surface B can be updated independently, but
> positioning it is still tied to the wl_surface A updates.
>
> (I'm not considering the corner cases arising from transitions from one
> commit mode to another at arbitrary times here, but they are solved
> for non-nested sub-surfaces.)
>
>
> Now, add a sub-sub-surface. Suppose we have a window consisting of the
> following:
>
> main surface, wl_surface A
>         ↳ sub-surface, wl_surface B, parent-cached
>                 ↳ sub-sub-surface, wl_surface C, parent-cached
>
> Assume the commit mode of all sub-surfaces is parent-cached, as
> written above. C.commit() will push C's pending state into C's cache.
> B.commit() will push B's pending state into B's cache. But does
> B.commit() do anything to C?
>
> As B is a sub-surface itself, needing a parent commit to change its
> current state, B.commit() certainly cannot make C's cached or pending
> state current.
>
> Suppose we have this sequence in protocol:
>
> C.commit(C1)
> B.commit(B1)
> C.commit(C2)
> A.commit(A1)
>
> What state should be current for each surface?
> For A, it is A1, of course.
> For B, it is B1, trivially.
> For C, should it be C1 which has been "blessed" by B, or C2 which is
> the latest but not "blessed" by B?
>
> If we say C1, then we will need a multi-level cache for each
> sub-surface, with a level of cache for each level of nesting.
> Sub-surface C would need a 2-level cache. C.commit() pushes pending
> state to C.cache-1. B.commit() pushes C.cache-1 to C.cache-2.
> A.commit() makes C.cache-2 as current state of C.
>
> Furthermore, each level of cache may hold a reference to a wl_buffer.
> This means, that the component driving wl_surface C will actually need
> a pool of 4 buffers:
> - a buffer that is currently on screen
> - a buffer that is in cache-2
> - a buffer that is in cache-1
> - a buffer that it is rendering to
>
> I do not think this is feasible. So, the answer must be that C2 is the
> current state for C. In other words, whether B.commit is called or not,
> it has no effect on what happens with C. (But does it make sense for
> applications?)

My first thought was "sure, C1 it is", but considering the huge cache
consumption if we have deeper sub-sub*-surface nesting, this really
doesn't make sense. So I was wondering whether C2 is actually that
bad.

So lets assume a user uses parent-cached sub-surfaces. This means,
they want the child to be updated only when the parent got updated.
This implicates, that the user has some kind of synchronization
between child and parent. Otherwise using parent-cached just doesn't
make sense as they could use the independent mode instead.

With this in mind, we know that if we have multi-layer nested
parent-cached sub-surfaces, then there already is a synchronization
between _all_ of them. So saying "parent-cached" means
"top-most-parent-cached" is actually fine for these use-cases, isn't
it?. Of course, "top-most-parent" means the top most parent with only
"parent-cached" in between.

Ok, but are there situations where parent-cached makes sense but there
is no synchronization in the client already?
I actually cannot think of anything. The thing is, if we have
parent-cached without synchronization in the client, the parent
surface does not know the current state of the client. So it has no
idea which frame the client currently has pending, it just knows that
it is only displayed _together_ with the parent.
So when the parent does the commit, it knows the client is updated
atomically, but it has no idea what frame the client actually drew. So
what is the reason that you actually want the atomic update in this
case? The independent mode would be just fine here. The only drawback
is that the parent no longer controls the frame-rate of the client.
But does that really matter?

So if we can conclude that the client does synchronization between
parent-cached subsurfaces, we can actually rename parent-cached to
top-most-parent-cached and be fine.
If anyone comes up with a use-case for _real_ parent-cached (that is,
bottom-most-parent-cached), we can always implement that with the
heavy caching, can't we?

And top-most-parent-cached and independent mode are both trivial to
implement, if I understand you correctly?

>
> Next a case with mixed commit modes:
>
> main surface, wl_surface A
>         ↳ sub-surface, wl_surface B, independent
>                 ↳ sub-sub-surface, wl_surface C, parent-cached
>
> C.commit() pushes C's pending state to C's cache.
> B.commit() make B's pending state current, and make C's cached state current, too.
> A.commit() make A's pending state current, and... does nothing to B,
> since B is independent. Also, does not even look past B, so nothing for C.
>
> Ok, I guess that works. And the case where B is parent-cached and C is
> independent should be obvious, too.
>
>
>
> This leads to the following commit algorithm:
>
> - Main surface commit, and sub-surface commit in independent mode, will
>   make own pending state current, and will call recursive commit on all
>   its immediate sub-surfaces.
>
> - Sub-surface commit in parent-cached mode will push the pending state
>   into the sub-surface's cache. No recursion.
>
> - A recursive commit on a sub-surface with independent commit mode does
>   nothing. No recursion.
>
> - A recursive commit on a sub-surface with parent-cached commit mode
>   will make the surface's cached state current, and call recursive
>   commit on all its immediate sub-surfaces.
>
> And then add all the corner cases when commit modes are changing, and
> e.g. a sub-surface is committed in independent commit mode with some
> state in its cache.
>
> Oh, and I forgot to specify what happens with
> wl_subsurface.set_position. D'oh.
>
>
> So, what are the benefits from nesting sub-surfaces?
> - Each sub-surface's position is relative to its parent, so you don't
>   have to add up the offsets in the client.
> - An independent sub-surface can have its own sub-surfaces, and update
>   the whole wl_surface sub-tree in sync.
> - ...?
>
>
> Is this all really worth it?
> Does anyone have a good use case for nested sub-surfaces, which would
> be inconvenient to implement with just sibling sub-surfaces without
> nesting?
>
> Unless there is a good use case I can design for, I'm tempted to just
> not bother with this, this time. If we later see, that nesting
> would actually have value, we can add the support by just bumping the
> wl_subcompositor interface version, and without any changes to the
> protocol signature.

Use-case for sub-sub-surfaces:

- flash-player embedded in webkit embedded in some application
I often see applications that embed a browser (for instance STEAM). If
that browser embeds other stuff, you want it to be independent of the
parent.
This use case requires top-most-parent-cached or independent. But I
cannot think of a reason why it requires bottom-most-parent-cached,
can you?

If we don't allow sub-surface nesting, then clients will have to
provide some sub-surface allocation forwarding. So for instance in my
use case described above, the application would provide a sub-surface
allocator to webkit, which would allocate the sub-surface for
flashplayer. But webkit would not be able to allocate the sub-surface
itself from the Wayland compositor as it itself is a sub-surface.
So in this case, we already have the synchronization of the clients
via the sub-surface allocator in the application. So there _is_ a
communication channel between the application and it's
sub-sub-sub-...-surfaces. This allows us to use top-most-parent-cached
instead of (bottom-most-)parent-cached.

=> Implement sub-surface nesting in the compositor to avoid having
each and every application to implement the sub-surface forwarding if
they themselves run as sub-surfaces.


Anyway, I really like the proposal. Apart from few typos the protocol
looks really good and I agree that we can implement nesting later.

Thanks
David