RFC: libei - emulated input in Wayland compositors

[Peter Hutterer]

On Fri, Jul 31, 2020 at 12:16:55PM +0300, Pekka Paalanen wrote:
> On Fri, 31 Jul 2020 15:13:50 +1000
> Peter Hutterer <peter.hutterer at who-t.net> wrote:
> 
> > I've been working on a new approach for allowing emulated input devices in
> > Wayland. Or in short - how can we make xdotool and synergy work? And
> > eventually replace them.
> > 
> > The proposal I have is a library for Emulated Input, in short libei.
> >   https://gitlab.freedesktop.org/whot/libei/
> > 
> > libei has two parts, the client side (libei) for applications and
> > a server side (libeis) for the compositor. The two libraries communicate
> > with each other (how? doesn't matter, it's an implementation detail) to
> > negotiate input devices.
> > 
> > The process is roughly:
> > - the libei client connects and says "I am org.freedesktop.SomeApplication
> >   and I want a pointer and a keyboard device"
> > - the libeis server says "ok, you can have a pointer device and a keyboard
> >   device"
> > - the libei client says 'move the pointer by 1/1', etc. and the server does
> >   just that. or not, depending on context.
> > 
> > There are more details, see the README in the repo and the libei.h and
> > libeis.h header files that describe the API.
> > 
> > The sticking point here is: emulated input comes via a separate channel.
> > The server a) knows it's emulated input, b) knows who it is coming from and
> > c) has complete control over the input.
> > 
> > a) is interesting because you can differ between the events internally. The
> > API right now is very similar to libinput's events so integrating it into a
> > compositor should be trivial.
> > 
> > b) is somewhat handwavy if an application runs outside a sandbox - any
> > information will be unreliable. Flatpak gives you an app-id though and
> > with that we can (eventually) do things like storing the allow/deny
> > decisions of the user in the portal implementation.
> > 
> > c) allows you to e.g. suspend the client when convenient or just ignore
> > certain sequences altogether. The two made-up examples are: suspend EI
> > during a password prompt, or allow EI from the software yubikey *only*
> > during a password prompt.
> > 
> > Now, the next question is: how do they *start* talking to each other?
> > libei provides multiple backends for the initial connection negotiation. My
> > goal is to have this work with flatpak portals so an application running
> > within the sandbox can be restricted accordingly. Alternatives to this could
> > be public DBus interfaces, simple fd passing or (as is implemented right
> > now) a named unix socket.
> > 
> > The aim is that a client can simply iterate through all of the options until
> > finds a connection. Once that's found, the actual code for emulating input is
> > always the same so it's trivial to implement a client that works on any
> > compositor that supports some backend of libeis.
> > The server part only needs to care about the negotiation mechanisms it
> > allows, i.e. GNOME will only have dbus/portal, sway will only have... dunno,
> > fd exchange maybe?
> > 
> > Next: because we have a separate channel for emulated input we can hook up
> > XTEST to use libei to talk to a compositor. I have a PoC implementation for
> > weston and Xwayland:
> >   https://gitlab.freedesktop.org/whot/weston/-/commits/wip/eis
> >   https://gitlab.freedesktop.org/whot/xserver/-/commits/wip/xwayland-eis
> > With that xdotool can move the pointer. Note this is truly the most minimal
> > code just to illustrate the point but you can fill in the blanks and do
> > things like the compositor preventing XTEST or not, etc.
> > 
> > This is all in very early stages with very little error checking so things
> > will probably crash or disconnect unexpectedly. I've tried to document the
> > API to make the intentions clear but there are still some very handwavy
> > bits.
> > 
> > Do let me know if you have any questions or suggestions please though.
> 
> Hi Peter,
> 
> this seems like a very well thought out proposal. I like it.
> Particularly the "how do you connect and get authorised" is a good
> compromise since there is no one solution fitting all.

Thanks!

> When an ei client creates a new emulated input device, how should
> Wayland compositors handle that wrt. wl_seats? Add it into the main
> wl_seat? Create a new wl_seat for each ei client? Is there enough
> information for a compositor to decide?

I think you should treat them like a physical input device, except that you
know they're emulated ones. So the policy will be compositor-specific and/or
client-specific (e.g.  it may make sense to give the software yubikey a
separate seat, if only to have a fixed layout).

But the key-value storage I mentioned in the other email could be used for
that. As for "is there enough information" it mostly comes down to: what do
you need to make that decision? If we know that, we can figure out where to
get it from.

> Is it possible to "close the loop" in a Wayland/ei client, to know
> which wl_seat is delivering the emulated input events? I'm not yet sure
> if that's even necessary, but I am thinking about implementing
> compositor input tests using ei. Do you see something missing from the
> ei API to work as the only API needed for writing compositor input
> tests?

I think the current plans for libei will be enough for input tests. Seat
assignment I haven't covered because I've not yet covered the full circle
(loop) yet - the input events are triggered externally and thus there's no
wl_seat.

I think there will be two parts to libei eventually, the client->server part
to generate events (xdotool) and the server->client part to capture events
(synology). This is possible with the relative pointer interface atm but
there might be room for something more specific. I'm hazy about those parts
though. That would closing the loop but we'd need triggers for this (e.g.
start capturing the pointer when the screen edge is hit) and I haven't
figured those out yet.

> Presumably there needs to be a known policy about when the compositor
> creates and removes wl_seats, but maybe that can be just a internal
> detail shared by the compositor and its test suite. Or maybe that could
> be explicit through other protocol that establishes the ei connection
> (fd passing), so that wl_seat creation is explicit.

I think a seat property is a good idea. Whether that's passed through libei
somehow or elsewhere - I simply don't know yet.

Cheers,
   Peter