Compositor handoffs: Switching clients between compositors

[Carsten Haitzler]

On Mon, 16 Aug 2021 14:13:06 +0100 David Edmundson <david at davidedmundson.co.uk>
said:

FYI we did this a few years back for efl and enlightenment... on a loss of the
compositor socket, toolkit retires connecting and comes back.

we added an extended protocol for the compositor to send a UUID to the client
per surface and clients on reconnect provide that UUID to the compositor - this
allows the compositor to fix all the stacking and other state when the surfaces
come back. :)

https://git.enlightenment.org/core/enlightenment.git/tree/src/protocol/session-recovery.xml

pretty simple - you set_uuid if you are recovering and set the previous uuid
you had. if not and its a fresh new surface you get_uuid to get the compositor
provided uuid and then store this for recovery purposes.

on a CLEAN destroy of your window you destroy_uuid(). these uuids can also be
used for full sessions (eg shut down machine, boot again and then have all your
windows opened up in their previous state - so compositor can store the uuid's
on disk as well as client apps too do the same).

you might want to give this a go and then all the rest of the state is solved.
i think in theory efl apps should then work with your hand-off (for us its
session recovery but that is your intent too - handling compositor crashes
and.or on the fly upgrades which require it to re-execute).

> Hi all,
> 
> I have been working on a method of "Compositor handoffs"--allowing
> clients to switch between compositors at runtime--and I wanted to
> share my progress with everyone and gather some feedback.
> 
> # Overview
> 
> The design is very simple at it's core. A client knows everything
> about the state of it's windows and still owns all appropriate memory.
> In the event of a compositor exit, the client should be able to just
> send everything again like it's a brand new client.
> 
> I have successfully ported: Qt, SDL and XWayland to do this, and they
> have worked with practically every application using those toolkits
> that I have thrown at it.
> 
> The primary goal of this is to handle a compositor crash. Whilst rare,
> it happens to some extent to all of the desktops and with wayland
> still being such a dynamic landscape and taking on more and more
> responsibilities for security reasons this isn't something that will
> end anytime soon. We want to make sure we don't have a path where
> users can lose data.
> 
> Long term this could solve other future goals, like switching between
> compositors across graphic cards, supporting CRIU (checkpoint restore
> in userspace) for GUI applications for mobile, or dynamically
> switching between the active compositor and waypipe. All of which will
> be really interesting new features.
> 
> It also makes the developer experience a billion times better, as I am
> no longer needing to log out to actually test changes, or only test in
> contrived nested situations.
> 
> # Slightly more detail
> 
> ## The compositor
> 
> The only change we needed to make in the compositor was making our
> public facing socket (i.e /var/run/.../wayland-0) persist on the file
> system. We run a small helper that creates the socket then spawns our
> real compositor. If our compositor crashes, we re-use the same socket.
> Clients still receive the same error about being disconnected, but
> there are no races should a client try to reconnect whilst a
> compositor is starting.
> 
> systemd's socket activation would also be an option here, though we
> ended up writing something custom.
> 
> ## Toolkit changes
> 
> Whilst this appears as an invasive thing to retroactively insert, it
> turns out to be relatively unintrusive and small. Most of the tasks we
> need to do involve functions that already exist anyway.
> 
> We need to:
>      - reset every wl_output (which can happen at runtime already)
>      - reset input devices (which can happen at runtime already)
>      - reset our data devices (which can get cancelled at runtime already)
>      - reset our wl_surface/xdg_shell objects
>      - send new contents (which happens already)
> 
> So for the most part we're just hooking into existing code.
> 
> ## SDL: https://github.com/davidedmundson/SDL/ branch reconnect
> 
> With the above patchset if the compositor crashes, all SDL apps will
> reconnect. This was tested against everything in SDL's test directory
> as well as obviously supertuxkart, the most important client on all
> our machines.
> 
> This is a good example to look at to see the extent of toolkit
> changes, the changes come in at around 200 lines, and the vast
> majority of that is in the cursor code.
> 
> ## Qt (Qt5 version)
> https://invent.kde.org/davidedmundson/qtwayland/-/tree/reconnect_main
> 
> Qt's usage is slightly  more complicated than SDL, but still the
> changes are still relatively small and managable.
> 
> I have been running this for the best part of 6 months, and everything
> "just works". I've thrown games, full IDE's (kdevelop) and image
> edtiors at this and everything works flawlessly without client
> changes.
> 
> This is slightly held back by us having to be API and ABI compatible
> with the frozen Qt5 base, a change landing in Qt6 could potentially
> address this cleaner, especially the part about refreshing the
> QBackingStore.
> 
> Note that if running Qt on Plasma we have a "plasma-integration"
> plugin that also makes some wayland calls that has also needed
> adjusting. The wiki
> (https://invent.kde.org/plasma/kwin/-/wikis/Restarting) lists all
> repositories and pending changes.
> 
> ## XWayland (somewhat WIP):
> https://gitlab.freedesktop.org/davidedmundson/xserver/-/tree/reconnections
> 
> Xwayland was a whole new adventure, but the code for resending windows
> was very simple even tackling details like pointer constraints. Being
> xwayland it even retains positional information as the X client itself
> remains intact.
> 
> The biggest challenge here is that the compositor typically starts
> Xwayland and does so with custom file descriptors for direct
> connections. That obviously doesn't work for handling the compositor
> changing. This required quite a lot of reshuffling about--moving
> XWwayland spawning to the wrapping helper, as well as adding new
> mutexes to handle startup being announced. Ultimately everything was
> do-able. Messy branches for that are available on request.
> 
> ## Other toolkits.
> 
> One nice thing is that it's completely opt-in on the client. For some
> system services (i.e plasmashell ) just restarting is a perfectly
> acceptable approach. For wl-paste, it may as well just exit. Firefox
> has such good in-built crash handling that I just wrapped the
> executable in a tiny wrapper and it works well enough. We don't need
> to have one solution fitting everything.
> 
> Ultimately the absolutely worst case is the application quits/crashes,
> which is exactly what happens now.
> 
> ## Handling openGL https://gitlab.freedesktop.org/davidedmundson/mesa/
> branch reconnect
> 
> OpenGL presented the biggest challenge. We only have the one
> EGLDisplay object persist for the lifespan of the application, we pass
> the wl_display constructor in as an argument, and therefore we need to
> persist the same wl_display object across connections.
> 
> This has meant we need to change both Mesa and libwayland:
> 
> When a client reconnects after an error a signal is emitted which is
> picked up by mesa. Any globals are then replaced, so new objects are
> created against the new connection.
> 
> With these changes the rest is easy. We need to recreate the EGLWindow
> against a new wl_surface on the new connection, but the context
> survives completely untouched and we can render a new frame right
> away.
> 
> ## libwayland changes:
> https://gitlab.freedesktop.org/davidedmundson/wayland/-/tree/reconnections
> 
> As mentioned above, we have a new method called "reconnect" and a
> signal that is emitted to all subscribers once reconnect is called. As
> it introduces new API all other changes rely on it.
> 
> Another big change here is handling of dangling objects. After a
> client reconnects all proxy objects are marked as "defunct" and we no
> longer send any requests on these objects, as the compositor is
> obviously unaware of these IDs. However calling the proxy's destructor
> needs to still work to free memory.  By doing this inside libwayland
> we are able to make it quite easy for toolkits to recreate objects
> "lazily". This has proved very useful when wayland proxies are hidden
> inside opaque pointers inside the toolkit, or if rendering is on
> another thread.
> 
> # Expected questions
> 
> ## Pics or it didn't happen
> 
> Here is a video of restoration working fully:
> https://home.davidedmundson.co.uk/index.php/s/KbQyb3eiBodTcFm
> 
> ## If the changes address only toolkits, what if I use low level
> wayland proxies in my apps?
> 
> The relevant change (watching for the signal on the wl_display and
> recreating a registry, new globals and new objects would need to be
> redone in the client. Within KDE, at least direct use of proxies
> outside the toolkit and a few select libraries is very very rare.
> 
> ## What happens to the clipboard
> 
> The current selection is lost, just like if the client closed. A
> clipboard manager can alleviate this. Every new selection afterwards
> behaves as expected.
> 
> ## What about window size/position/stacking order
> 
> Because everything is treated as new windows (from a compositor POV)
> stacking order and positioning is effectively random. Size is often
> maintained as the clients do know their original size. Session
> managment
> (https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/18)
> is a possible solution to this.
> 
> ## Could it be done without toolkit changes via a magic proxy?
> 
> Possibly, but I don't think it adds anything. Clients are the
> canonical source of what a client needs, and knows how to handle any
> sudden changes the best. There is also zero overhead this way.
> 
> ## What if the EGLDisplay capabilities change?
> 
> My mesa change keeps the same connection to the card specified by the
> first wl_drm.device event from the first wayland connection. If the
> new compositor passes a different device it is ignored.
> That definitely has some theoretical quirks, but none that have come
> up in practice for the immediate task at hand.
> 
> ## What if the compositor has no support?
> 
> Then the socket on the file system will disappear and clients will
> fail to reconnect and quit. We also use this in clients to detect a
> graceful kwin exit and just exit accordingly.
> 
> ## What about vulkan?
> 
> It will need similar changes to what we've done for OpenGL.
> 
> ## That was too much text
> 
> There will be an XDC talk
> (https://indico.freedesktop.org/event/1/contributions/20/) where I
> will describe everything mentioned here.
> 
> # Next steps
> 
> The changes to libwayland is the most invasive, and the most
> potentially controversial. I wanted to send an overview email and get
> a discussion going before I submitted merge requests. Unfortunately
> it's the blocker on everything else.
> 


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
Carsten Haitzler - raster at rasterman.com