RFC: libei - emulated input in Wayland compositors

[Peter Hutterer]

An update to the work that's been happening here.

libei now has some xdg-desktop-portal support. It will ping the portal, get
the fd back and then go on from there. For Xwayland switching to use the a
portal is a one-line change, so a sandboxed X application now has XTEST
requests go through the portal.

The portal implementation is currently very simple and for initial
negotiation only, once the fd has been exchanged, the portal has no further
say in the matter. I suspect that's likely to change one way or another.

Olivier has been working on the Xwayland support, after some
initial testing we figured it's *not* possible to have libei support batch
requests directly. This is required for xdotool which fires off all
requests and exits immediately. Olivier has added event buffering
so we can replay the events once the device negotiation is done.
He's also added the bits so we can "identify" clients based on their X
client name.

There's a new part to manage Restrictions for EIS, libreis. The portal can
use that on the fd to limit what the client will be able to do later. Right
now the portal sets the client's name to the app-id (and that name cannot be
changed by the client, this also applies to Xwayland). It's early stages
otherwise, lock down of capabilities (e.g. deny keyboard caps) is
mostly just sketched out, not tested.

I've been filling in the libei/libeis pieces though a lot of it is still in
the proof-of-concept stage. Do not run your nuclear fusion plant on this.
Either way, I reckon the whole thing looks promising.

Combination of branches for portal support is outlined here:
   https://gitlab.freedesktop.org/whot/libei/-/issues/1
Comes with free occasional crashes.
 
Cheers,
   Peter


On Fri, Jul 31, 2020 at 03:13:50PM +1000, Peter Hutterer wrote:
> I've been working on a new approach for allowing emulated input devices in
> Wayland. Or in short - how can we make xdotool and synergy work? And
> eventually replace them.
> 
> The proposal I have is a library for Emulated Input, in short libei.
>   https://gitlab.freedesktop.org/whot/libei/
> 
> libei has two parts, the client side (libei) for applications and
> a server side (libeis) for the compositor. The two libraries communicate
> with each other (how? doesn't matter, it's an implementation detail) to
> negotiate input devices.
> 
> The process is roughly:
> - the libei client connects and says "I am org.freedesktop.SomeApplication
>   and I want a pointer and a keyboard device"
> - the libeis server says "ok, you can have a pointer device and a keyboard
>   device"
> - the libei client says 'move the pointer by 1/1', etc. and the server does
>   just that. or not, depending on context.
> 
> There are more details, see the README in the repo and the libei.h and
> libeis.h header files that describe the API.
> 
> The sticking point here is: emulated input comes via a separate channel.
> The server a) knows it's emulated input, b) knows who it is coming from and
> c) has complete control over the input.
> 
> a) is interesting because you can differ between the events internally. The
> API right now is very similar to libinput's events so integrating it into a
> compositor should be trivial.
> 
> b) is somewhat handwavy if an application runs outside a sandbox - any
> information will be unreliable. Flatpak gives you an app-id though and
> with that we can (eventually) do things like storing the allow/deny
> decisions of the user in the portal implementation.
> 
> c) allows you to e.g. suspend the client when convenient or just ignore
> certain sequences altogether. The two made-up examples are: suspend EI
> during a password prompt, or allow EI from the software yubikey *only*
> during a password prompt.
> 
> Now, the next question is: how do they *start* talking to each other?
> libei provides multiple backends for the initial connection negotiation. My
> goal is to have this work with flatpak portals so an application running
> within the sandbox can be restricted accordingly. Alternatives to this could
> be public DBus interfaces, simple fd passing or (as is implemented right
> now) a named unix socket.
> 
> The aim is that a client can simply iterate through all of the options until
> finds a connection. Once that's found, the actual code for emulating input is
> always the same so it's trivial to implement a client that works on any
> compositor that supports some backend of libeis.
> The server part only needs to care about the negotiation mechanisms it
> allows, i.e. GNOME will only have dbus/portal, sway will only have... dunno,
> fd exchange maybe?
> 
> Next: because we have a separate channel for emulated input we can hook up
> XTEST to use libei to talk to a compositor. I have a PoC implementation for
> weston and Xwayland:
>   https://gitlab.freedesktop.org/whot/weston/-/commits/wip/eis
>   https://gitlab.freedesktop.org/whot/xserver/-/commits/wip/xwayland-eis
> With that xdotool can move the pointer. Note this is truly the most minimal
> code just to illustrate the point but you can fill in the blanks and do
> things like the compositor preventing XTEST or not, etc.
> 
> This is all in very early stages with very little error checking so things
> will probably crash or disconnect unexpectedly. I've tried to document the
> API to make the intentions clear but there are still some very handwavy
> bits.
> 
> Do let me know if you have any questions or suggestions please though.
> 
> Cheers,
>   Peter