[RFC wayland-protocols] inputfd - direct input access protocol

[Roderick Colenbrander]

Hi Bastien,


On Tue, Apr 4, 2017 at 2:17 AM, Bastien Nocera <hadess at hadess.net> wrote:
> Hey Roderick,
>
> On Mon, 2017-04-03 at 16:08 -0700, Roderick Colenbrander wrote:
>> Hi Peter,
>>
>> Thanks for sharing this proposal. I have some little comments for a
>> later stage, but would rather discuss some big items first.
>>
>> The feedback I will share comes from a couple different angles as we
>> are interested in Wayland for various different use cases.
>>
>> Originally we had the gamepad wayland thread originally started by
>> Jingkui. The proposal there was to provide knowledge of axes, buttons
>> and everything making up a gamepad to the compositor. He proposed
>> this direction as a way to inject events from Chrome into Android as
>> at that level there is no file descriptor anymore. Ourselves we were
>> quite interested in this approach as well. Among the things we do is
>> roughly remote streamable desktops. Currently on X you can inject
>> keyboard/mouse events through XTest, but the same applies to a custom
>> Wayland compositor into which you can easily inject keyboard/mouse
>> data. Clients don't have to be aware of the custom virtual devices
>> and protocols behind the scenes.
>
> The main reason why you wouldn't want to use a custom protocol is that
> we haven't been able to add support for joypad events in X11 in nearly
> 30 years, and, on Linux, the "evdev" joystick API offered by the kernel
> matched pretty well the requirements from application/game developers.
>
> We could add a layer on top of evdev, but we'd end up copying all of
> evdev's features. The reason why we want an API in Wayland to pass the
> file descriptor around is:
> - we want only the focused app to have access to the joystick, this
> makes it easier to have multiple "apps" access the joystick
> - we want to be able to discover and receive events when new joysticks
> become available, without access to udev or equivalent as we'd be
> sandboxed
>
> With that said...
>
>> When using inputfd for such a use case, either we would need to fake
>> an input device through uinput (not ideal) or add a new 'fd_type' and
>> have a custom protocol across this. Clients or input libraries would
>> need to be able to deal with this other protocol, which would be a
>> big pain. For such remote desktop use cases and even the Android one
>> inputfd is not ideal.
>
> I don't quite understand which parts of this system are in your
> control, and which ones aren't, what is generic, and what is custom.
> I'm guessing that the OS and compositor (Chrome OS, or Android) have
> access to the raw device which is available locally, but the (custom)
> application would send events through the network to the remote
> application. Is this correct?

That's correct. Just for the record I squeezed 2 use cases together.
The Android/ChromeOS is one and the remote application is another. In
the remote application we are in control of protocol and remote
compositor, some sandbox, the system libraries on the remote side for
graphics, audio, input etcetera ideally are unmodified. If doing
inputfd passing, we would need to modify SDL and whatever other
libraries. I realize this is quite a custom use case, but in one way
not too different from various automated testing and remote control
use cases which currently use XTest in case of X11.

> For Linux desktops, I'd expect toolkits (SDL and GTK+) to be first in
> the list of modified code to handle the new protocol. In SDL's case,
> this replaces the udev backend code for joysticks.
>
>>
> <snip>
>> The current proposal could work for exposing basic gamepad
>> functionality for the axes/buttons.
>
> Not having a layer in between the kernel and the app means that adding
> new features in the kernel and supporting them in the app requires just
> two source trees to be modified. The upstream evdev API *is* the API.
>
>>  Taking hid-sony as an example ds3 and ds4 devices are now threated
>> as composite devices utilizing multiple device nodes. The ds4 is
>> probably the best example using 3 evdev nodes, 1 for gamepad, 1 for
>> touchpad (reported as a pointer device) and 1 for motion sensors. I
>> expect the Switch Pro controller and the Joypads for the Switch to
>> use at minimum 2 nodes as well.
>
> I recently added support for the PS3 uDraw tablet, it also has a
> drawing tablet node on top of the 3 you mentioned above ;)
>
> Peter mentioned the device group identifiers to be able to tie them
> together.
>
>> How would composite devices be handled? There would need to be a way
>> to tie multiple devices together. So far composite devices use all
>> the same ids (product/vendor) and same unique id (EVIOCGUNIQ) and
>> even physical location (EVIOCGPHYS). At minimum I think the
>> compositor needs to tie these together with some sort of shared id.
>
> The compositor is also outside a sandbox, and can tie them together as
> those devices have the same sysfs parent device.
>
>> Another issue again taking ds4 as an example it has a touchpad, which
>> is already picked up as a pointer device elsewhere. How to handle
>> this? Unflag it as a pointer and share this inputfd as well or use it
>> really as a pointer and gamepad state the other way?
>
> How do you access it now, and stop the OS from handling it as a system
> pointer?

We just fixed the hid-sony driver for 4.12 previous didn't advertize
itself properly yet. Before 4.10 touchpad and gaming where crammed
into one device and they were stepping on each others toes, so we
haven't really dealt with this properly yet. Users of 4.12 would now
see the ds4 as a pointer, so it may need some consideration soon.

> Do you set up a grab, and collect the events as passed through
> the Wayland protocol, or would you prefer having raw access to the
> device node?

Honestly I don't know what the best would be, we are not doing
anything just yet. I guess it will heavily depend on the use case.
Specific to our potential use case we wouldn't need this capability,
but others might. I can imagine someone liking pointer capability for
Kodi on a TV if the user controller Kodi with a DS4 or even Steam
Controller.

I wonder how Valve is doing this on Steam as with the Steam
Controller, which I haven't used. I think they are allowing it to
replace keyboard and mouse kind of. Curious how they are solving this.
My guess is they are actually spoofing a mouse for game compatibility.

>
>> In addition next to evdev nodes there are sysfs nodes for which fd
>> passing won't work. The best example would be LEDs as found on
>> ds3/ds4/xbox360, but I'm sure there will be GPIO as well. In case of
>> ds4 it is not uncommon to update the LEDs many times a second (often
>> used for light effects). How would such features be exposed? Is the
>> idea to leverage properties or something like this? Passing these
>> nodes to clients is probably not a good idea.
>
> I'd expect the LEDs to be handled by the OS, or the protocol to be
> enhanced to be able to set those.

In our case we want the applications to be able to manage the LEDs. In
some cases it could make sense for a compositor (e.g. more SteamOS
like) to manage player numbers. If you look at the DS4 and actual
games they do color animations on the device. For example green may be
alive, while flashing orange may mean you are in danger and red you
died. This is game managed.

>
>> This is some initial feedback. I'm not sure how much I like the
>> inputfd proposal. For a part because it doesn't fit some of our use
>> cases well (remote desktop + custom protocols). In addition I'm also
>> a bit worried about the complexity in handling composite devices.
>
> I think it's actually much better for you. The original protocol was
> another protocol on top of the standard one for Linux ("evdev"), it
> crippled the abilities offered by the kernel (no rumble, no touchpad,
> no accelerometer, no gyro, no LEDs, no separate rudders, etc.). It
> barely handled an "XBox 360" level controller, and was definitely
> cutting functionality from even a PS3 controller, let alone a PS4 one.
>
> It also had absolutely no support for handling composite devices. This
> proposal was just about good enough to implement the gamepad HTML5 API,
> it was in no way good enough to handle all the cases offered by native
> games.

I'm quite familiar with the original spec. Offline I had a lot of
conversations with Jingkui about it. We had ideas to add all these
missing areas, but it would have become more evdev like. We didn't
have time yet to work on a more detailed spec.


>
> Let me know whether I forgot something, or have any other questions.
>
> Cheers