RFC: libei - emulated input in Wayland compositors

[Pekka Paalanen]

On Fri, 31 Jul 2020 15:13:50 +1000
Peter Hutterer <peter.hutterer at who-t.net> wrote:

> I've been working on a new approach for allowing emulated input devices in
> Wayland. Or in short - how can we make xdotool and synergy work? And
> eventually replace them.
> 
> The proposal I have is a library for Emulated Input, in short libei.
>   https://gitlab.freedesktop.org/whot/libei/
> 
> libei has two parts, the client side (libei) for applications and
> a server side (libeis) for the compositor. The two libraries communicate
> with each other (how? doesn't matter, it's an implementation detail) to
> negotiate input devices.
> 
> The process is roughly:
> - the libei client connects and says "I am org.freedesktop.SomeApplication
>   and I want a pointer and a keyboard device"
> - the libeis server says "ok, you can have a pointer device and a keyboard
>   device"
> - the libei client says 'move the pointer by 1/1', etc. and the server does
>   just that. or not, depending on context.
> 
> There are more details, see the README in the repo and the libei.h and
> libeis.h header files that describe the API.
> 
> The sticking point here is: emulated input comes via a separate channel.
> The server a) knows it's emulated input, b) knows who it is coming from and
> c) has complete control over the input.
> 
> a) is interesting because you can differ between the events internally. The
> API right now is very similar to libinput's events so integrating it into a
> compositor should be trivial.
> 
> b) is somewhat handwavy if an application runs outside a sandbox - any
> information will be unreliable. Flatpak gives you an app-id though and
> with that we can (eventually) do things like storing the allow/deny
> decisions of the user in the portal implementation.
> 
> c) allows you to e.g. suspend the client when convenient or just ignore
> certain sequences altogether. The two made-up examples are: suspend EI
> during a password prompt, or allow EI from the software yubikey *only*
> during a password prompt.
> 
> Now, the next question is: how do they *start* talking to each other?
> libei provides multiple backends for the initial connection negotiation. My
> goal is to have this work with flatpak portals so an application running
> within the sandbox can be restricted accordingly. Alternatives to this could
> be public DBus interfaces, simple fd passing or (as is implemented right
> now) a named unix socket.
> 
> The aim is that a client can simply iterate through all of the options until
> finds a connection. Once that's found, the actual code for emulating input is
> always the same so it's trivial to implement a client that works on any
> compositor that supports some backend of libeis.
> The server part only needs to care about the negotiation mechanisms it
> allows, i.e. GNOME will only have dbus/portal, sway will only have... dunno,
> fd exchange maybe?
> 
> Next: because we have a separate channel for emulated input we can hook up
> XTEST to use libei to talk to a compositor. I have a PoC implementation for
> weston and Xwayland:
>   https://gitlab.freedesktop.org/whot/weston/-/commits/wip/eis
>   https://gitlab.freedesktop.org/whot/xserver/-/commits/wip/xwayland-eis
> With that xdotool can move the pointer. Note this is truly the most minimal
> code just to illustrate the point but you can fill in the blanks and do
> things like the compositor preventing XTEST or not, etc.
> 
> This is all in very early stages with very little error checking so things
> will probably crash or disconnect unexpectedly. I've tried to document the
> API to make the intentions clear but there are still some very handwavy
> bits.
> 
> Do let me know if you have any questions or suggestions please though.

Hi Peter,

this seems like a very well thought out proposal. I like it.
Particularly the "how do you connect and get authorised" is a good
compromise since there is no one solution fitting all.

When an ei client creates a new emulated input device, how should
Wayland compositors handle that wrt. wl_seats? Add it into the main
wl_seat? Create a new wl_seat for each ei client? Is there enough
information for a compositor to decide?

Is it possible to "close the loop" in a Wayland/ei client, to know
which wl_seat is delivering the emulated input events? I'm not yet sure
if that's even necessary, but I am thinking about implementing
compositor input tests using ei. Do you see something missing from the
ei API to work as the only API needed for writing compositor input
tests?

Presumably there needs to be a known policy about when the compositor
creates and removes wl_seats, but maybe that can be just a internal
detail shared by the compositor and its test suite. Or maybe that could
be explicit through other protocol that establishes the ei connection
(fd passing), so that wl_seat creation is explicit.


Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/wayland-devel/attachments/20200731/049528a7/attachment.sig>