RFC: libei - emulated input in Wayland compositors

[Roman Gilg]

On Fri, Jul 31, 2020 at 7:13 AM Peter Hutterer <peter.hutterer at who-t.net> wrote:
>
> I've been working on a new approach for allowing emulated input devices in
> Wayland. Or in short - how can we make xdotool and synergy work? And
> eventually replace them.
>
> The proposal I have is a library for Emulated Input, in short libei.
>   https://gitlab.freedesktop.org/whot/libei/

We talked about it already yesterday but thanks again for this great
project. I decided to directly write some experimental integration
code based on your Weston branch for the server library in KWinFT [1]
in order to try this out as a solution for my Steam Controller issue
[2] that - I assume - motivated the creation of this library to some
extent.

And yes, it works. :) I can move the cursor with the Steam controller
as in "Steam client -> XTEST -> patched Xwayland -> libei -> libeis ->
KWinFT" just fine.

Am I right in assuming that the button-press event is not yet done in
libei or in the patched Xwayland version you linked? When it's
available let me know and I'll add the necessary logic for that too.

> libei has two parts, the client side (libei) for applications and
> a server side (libeis) for the compositor. The two libraries communicate
> with each other (how? doesn't matter, it's an implementation detail) to
> negotiate input devices.
>
> The process is roughly:
> - the libei client connects and says "I am org.freedesktop.SomeApplication
>   and I want a pointer and a keyboard device"
> - the libeis server says "ok, you can have a pointer device and a keyboard
>   device"
> - the libei client says 'move the pointer by 1/1', etc. and the server does
>   just that. or not, depending on context.
>
> There are more details, see the README in the repo and the libei.h and
> libeis.h header files that describe the API.
>
> The sticking point here is: emulated input comes via a separate channel.
> The server a) knows it's emulated input, b) knows who it is coming from and
> c) has complete control over the input.
>
> a) is interesting because you can differ between the events internally. The
> API right now is very similar to libinput's events so integrating it into a
> compositor should be trivial.
>
> b) is somewhat handwavy if an application runs outside a sandbox - any
> information will be unreliable. Flatpak gives you an app-id though and
> with that we can (eventually) do things like storing the allow/deny
> decisions of the user in the portal implementation.
>
> c) allows you to e.g. suspend the client when convenient or just ignore
> certain sequences altogether. The two made-up examples are: suspend EI
> during a password prompt, or allow EI from the software yubikey *only*
> during a password prompt.
>
> Now, the next question is: how do they *start* talking to each other?
> libei provides multiple backends for the initial connection negotiation. My
> goal is to have this work with flatpak portals so an application running
> within the sandbox can be restricted accordingly. Alternatives to this could
> be public DBus interfaces, simple fd passing or (as is implemented right
> now) a named unix socket.

Wiring this somehow through portals would be important for sure.
Xwayland as a client could either be accepted by default or if
Olivier's Xwayland xdg-portal patches [3] land (with the additional
portal for libei) only be accepted after the user confirmed it just
like every other sandboxed client.

That being said the envisioned permission model is still somewhat
difficult for me to grasp. To reiterate: the access of sandboxed
clients can be accepted or rejected by the user. But to my
understanding that's a function of the xdg-portal itself. You said the
compositor can filter requests too. Can it only allow libei
connections through xdg-portals and Xwayland? What about other
clients, how can they be distinguished from xdg-portals and Xwayland
securely? Or is this only possible for flatpaked clients? Or is such a
client blocked from trying to do that anyway (in other words is it
allowed or not to connect to arbitrary sockets like the libei one)?

As it is probably clear now the overall concept of xdg-portals in
detail is still not very well understood by me. From conversations I
had lately with other windowing system developers I believe I'm not
the only one.

Since xdg-portals become more and more important for securing our
graphical sessions it would be great if someone with more knowledge
about it could create some kind of article or documentation about it
that looks at it from the perspective of windowing systems . How do
apps in/out of Flatpaks that display their pixels through X11,
Xwayland or Wayland directly work in respect to the sandboxed
environment provided by xdg-portals? What does this mean for a Wayland
compositor, what does it need to do or refrain from to be on the safe
side?

For example some simple but lucid diagrams like the one in libei's
README describing the flow around client <-> xdg-portals <-> windowing
system would probably already help many of us. If somebody feels
motivated to do that I would be happy to help, ping me on IRC.

[1] https://gitlab.com/romangg/kwinft/-/commits/libei
[2] https://gitlab.freedesktop.org/xorg/xserver/-/merge_requests/431
[3] https://gitlab.freedesktop.org/xorg/xserver/-/merge_requests/465

> The aim is that a client can simply iterate through all of the options until
> finds a connection. Once that's found, the actual code for emulating input is
> always the same so it's trivial to implement a client that works on any
> compositor that supports some backend of libeis.
> The server part only needs to care about the negotiation mechanisms it
> allows, i.e. GNOME will only have dbus/portal, sway will only have... dunno,
> fd exchange maybe?
>
> Next: because we have a separate channel for emulated input we can hook up
> XTEST to use libei to talk to a compositor. I have a PoC implementation for
> weston and Xwayland:
>   https://gitlab.freedesktop.org/whot/weston/-/commits/wip/eis
>   https://gitlab.freedesktop.org/whot/xserver/-/commits/wip/xwayland-eis
> With that xdotool can move the pointer. Note this is truly the most minimal
> code just to illustrate the point but you can fill in the blanks and do
> things like the compositor preventing XTEST or not, etc.
>
> This is all in very early stages with very little error checking so things
> will probably crash or disconnect unexpectedly. I've tried to document the
> API to make the intentions clear but there are still some very handwavy
> bits.
>
> Do let me know if you have any questions or suggestions please though.
>
> Cheers,
>   Peter
> _______________________________________________
> wayland-devel mailing list
> wayland-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/wayland-devel