Nesting sub-surfaces?

[Pekka Paalanen]

On Wed, 6 Mar 2013 15:01:38 +0100
David Herrmann <dh.herrmann at gmail.com> wrote:

> Hi Pekka
> 
> On Wed, Mar 6, 2013 at 1:49 PM, Pekka Paalanen <ppaalanen at gmail.com> wrote:
> > Hi,
> >
> > I have been pondering about sub-surface nesting, i.e. allowing
> > sub-sub-surfaces and so on, and I have hard time figuring out how
> > commits should work.
> >
> > The v2 sub-surface protocol proposal is here:
> > http://cgit.collabora.com/git/user/pq/weston.git/tree/protocol/subsurface.xml?h=subsurface-v2
> >
> > Adding support for nesting does not require any API changes to the
> > protocol. We just need to define, that it is ok to have a sub-surface
> > as a parent for another sub-surface, and how commits work. Commit
> > behaviour is again the hard part.
> >
> > Suppose we have a window consisting of the following:
> >
> > main surface, wl_surface A
> >         ↳ sub-surface, wl_surface B
> >
> > Every surface has current state (how it is currently drawn on screen),
> > and pending state (to accumulate pieces of state, that will then be
> > made current atomically on wl_surface.commit).
> >
> > Additionally, sub-surfaces have a state cache. When a sub-surface
> > itself is explicitly committed, the pending state either becomes
> > current, or goes to the cache, depending on the commit mode of the
> > sub-surface. The pending state can only be used when a commit request
> > was received for the surface itself, never via parent commit.
> >
> > If the commit mode for the sub-surface B is parent-cached, then
> > B.commit() will push the pending state into the cache. The cached state
> > is made current on A.commit(), so that all surfaces get updated
> > atomically.
> >
> > If the commit mode for the sub-surface B is independent, then
> > B.commit() will make the pending state current. A.commit() does not
> > affect B's current state, except for the wl_subsurface.set_position.
> >
> > The position set by a wl_subsurface.set_position request is always
> > stored, and applied only on parent surface commit (A.commit()). This
> > way the contents of wl_surface B can be updated independently, but
> > positioning it is still tied to the wl_surface A updates.
> >
> > (I'm not considering the corner cases arising from transitions from one
> > commit mode to another at arbitrary times here, but they are solved
> > for non-nested sub-surfaces.)
> >
> >
> > Now, add a sub-sub-surface. Suppose we have a window consisting of the
> > following:
> >
> > main surface, wl_surface A
> >         ↳ sub-surface, wl_surface B, parent-cached
> >                 ↳ sub-sub-surface, wl_surface C, parent-cached
> >
> > Assume the commit mode of all sub-surfaces is parent-cached, as
> > written above. C.commit() will push C's pending state into C's cache.
> > B.commit() will push B's pending state into B's cache. But does
> > B.commit() do anything to C?
> >
> > As B is a sub-surface itself, needing a parent commit to change its
> > current state, B.commit() certainly cannot make C's cached or pending
> > state current.
> >
> > Suppose we have this sequence in protocol:
> >
> > C.commit(C1)
> > B.commit(B1)
> > C.commit(C2)
> > A.commit(A1)
> >
> > What state should be current for each surface?
> > For A, it is A1, of course.
> > For B, it is B1, trivially.
> > For C, should it be C1 which has been "blessed" by B, or C2 which is
> > the latest but not "blessed" by B?
> >
> > If we say C1, then we will need a multi-level cache for each
> > sub-surface, with a level of cache for each level of nesting.
> > Sub-surface C would need a 2-level cache. C.commit() pushes pending
> > state to C.cache-1. B.commit() pushes C.cache-1 to C.cache-2.
> > A.commit() makes C.cache-2 as current state of C.
> >
> > Furthermore, each level of cache may hold a reference to a wl_buffer.
> > This means, that the component driving wl_surface C will actually need
> > a pool of 4 buffers:
> > - a buffer that is currently on screen
> > - a buffer that is in cache-2
> > - a buffer that is in cache-1
> > - a buffer that it is rendering to
> >
> > I do not think this is feasible. So, the answer must be that C2 is the
> > current state for C. In other words, whether B.commit is called or not,
> > it has no effect on what happens with C. (But does it make sense for
> > applications?)
> 
> My first thought was "sure, C1 it is", but considering the huge cache
> consumption if we have deeper sub-sub*-surface nesting, this really
> doesn't make sense. So I was wondering whether C2 is actually that
> bad.
> 
> So lets assume a user uses parent-cached sub-surfaces. This means,
> they want the child to be updated only when the parent got updated.
> This implicates, that the user has some kind of synchronization
> between child and parent. Otherwise using parent-cached just doesn't
> make sense as they could use the independent mode instead.
> 
> With this in mind, we know that if we have multi-layer nested
> parent-cached sub-surfaces, then there already is a synchronization
> between _all_ of them. So saying "parent-cached" means
> "top-most-parent-cached" is actually fine for these use-cases, isn't
> it?. Of course, "top-most-parent" means the top most parent with only
> "parent-cached" in between.
> 
> Ok, but are there situations where parent-cached makes sense but there
> is no synchronization in the client already?
> I actually cannot think of anything. The thing is, if we have
> parent-cached without synchronization in the client, the parent
> surface does not know the current state of the client. So it has no
> idea which frame the client currently has pending, it just knows that
> it is only displayed _together_ with the parent.
> So when the parent does the commit, it knows the client is updated
> atomically, but it has no idea what frame the client actually drew. So
> what is the reason that you actually want the atomic update in this
> case? The independent mode would be just fine here. The only drawback
> is that the parent no longer controls the frame-rate of the client.
> But does that really matter?
> 
> So if we can conclude that the client does synchronization between
> parent-cached subsurfaces, we can actually rename parent-cached to
> top-most-parent-cached and be fine.
> If anyone comes up with a use-case for _real_ parent-cached (that is,
> bottom-most-parent-cached), we can always implement that with the
> heavy caching, can't we?
> 
> And top-most-parent-cached and independent mode are both trivial to
> implement, if I understand you correctly?
> 
> >
> > Next a case with mixed commit modes:
> >
> > main surface, wl_surface A
> >         ↳ sub-surface, wl_surface B, independent
> >                 ↳ sub-sub-surface, wl_surface C, parent-cached
> >
> > C.commit() pushes C's pending state to C's cache.
> > B.commit() make B's pending state current, and make C's cached state current, too.
> > A.commit() make A's pending state current, and... does nothing to B,
> > since B is independent. Also, does not even look past B, so nothing for C.
> >
> > Ok, I guess that works. And the case where B is parent-cached and C is
> > independent should be obvious, too.
> >
> >
> >
> > This leads to the following commit algorithm:
> >
> > - Main surface commit, and sub-surface commit in independent mode, will
> >   make own pending state current, and will call recursive commit on all
> >   its immediate sub-surfaces.
> >
> > - Sub-surface commit in parent-cached mode will push the pending state
> >   into the sub-surface's cache. No recursion.
> >
> > - A recursive commit on a sub-surface with independent commit mode does
> >   nothing. No recursion.
> >
> > - A recursive commit on a sub-surface with parent-cached commit mode
> >   will make the surface's cached state current, and call recursive
> >   commit on all its immediate sub-surfaces.
> >
> > And then add all the corner cases when commit modes are changing, and
> > e.g. a sub-surface is committed in independent commit mode with some
> > state in its cache.
> >
> > Oh, and I forgot to specify what happens with
> > wl_subsurface.set_position. D'oh.
> >
> >
> > So, what are the benefits from nesting sub-surfaces?
> > - Each sub-surface's position is relative to its parent, so you don't
> >   have to add up the offsets in the client.
> > - An independent sub-surface can have its own sub-surfaces, and update
> >   the whole wl_surface sub-tree in sync.
> > - ...?
> >
> >
> > Is this all really worth it?
> > Does anyone have a good use case for nested sub-surfaces, which would
> > be inconvenient to implement with just sibling sub-surfaces without
> > nesting?
> >
> > Unless there is a good use case I can design for, I'm tempted to just
> > not bother with this, this time. If we later see, that nesting
> > would actually have value, we can add the support by just bumping the
> > wl_subcompositor interface version, and without any changes to the
> > protocol signature.
> 
> Use-case for sub-sub-surfaces:
> 
> - flash-player embedded in webkit embedded in some application
> I often see applications that embed a browser (for instance STEAM). If
> that browser embeds other stuff, you want it to be independent of the
> parent.
> This use case requires top-most-parent-cached or independent. But I
> cannot think of a reason why it requires bottom-most-parent-cached,
> can you?
> 
> If we don't allow sub-surface nesting, then clients will have to
> provide some sub-surface allocation forwarding. So for instance in my
> use case described above, the application would provide a sub-surface
> allocator to webkit, which would allocate the sub-surface for
> flashplayer. But webkit would not be able to allocate the sub-surface
> itself from the Wayland compositor as it itself is a sub-surface.
> So in this case, we already have the synchronization of the clients
> via the sub-surface allocator in the application. So there _is_ a
> communication channel between the application and it's
> sub-sub-sub-...-surfaces. This allows us to use top-most-parent-cached
> instead of (bottom-most-)parent-cached.
> 
> => Implement sub-surface nesting in the compositor to avoid having
> each and every application to implement the sub-surface forwarding if
> they themselves run as sub-surfaces.
> 
> 
> Anyway, I really like the proposal. Apart from few typos the protocol
> looks really good and I agree that we can implement nesting later.

Thank you, and thank you all for the discussion on #wayland, it really
gave me some new ideas. I'm not replying to your specific points right
now, David, since I have to re-evaluate the design now.

I realized the wl_subsurface.set_commit_mode() is not right, if we have
nesting. It cannot be per-surface, but it must affect the whole
sub-tree of sub-surfaces for atomic resizing to work.

That might actually solve a lot of weirdness there.

And I was certainly wrong in thinking that adding nesting support as an
afterthought would be anything but ugly.

So, stay tuned! ;-D


Thanks,
pq