[RFC wayland-protocols] inputfd - direct input access protocol
Roderick Colenbrander
roderick at gaikai.com
Wed Apr 5 23:51:51 UTC 2017
On Mon, Apr 3, 2017 at 9:08 PM, Peter Hutterer <peter.hutterer at who-t.net> wrote:
>
> On Mon, Apr 03, 2017 at 04:08:21PM -0700, Roderick Colenbrander wrote:
> > Hi Peter,
> >
> > Thanks for sharing this proposal. I have some little comments for a later
> > stage, but would rather discuss some big items first.
> >
> > The feedback I will share comes from a couple different angles as we are
> > interested in Wayland for various different use cases.
>
> good :) that's exactly what we need right now to get this right.
>
> > Originally we had the gamepad wayland thread originally started by Jingkui.
> > The proposal there was to provide knowledge of axes, buttons and everything
> > making up a gamepad to the compositor. He proposed this direction as a way
> > to inject events from Chrome into Android as at that level there is no file
> > descriptor anymore. Ourselves we were quite interested in this approach as
> > well. Among the things we do is roughly remote streamable desktops.
> > Currently on X you can inject keyboard/mouse events through XTest, but the
> > same applies to a custom Wayland compositor into which you can easily
> > inject keyboard/mouse data. Clients don't have to be aware of the custom
> > virtual devices and protocols behind the scenes.
> >
> > When using inputfd for such a use case, either we would need to fake an
> > input device through uinput (not ideal) or add a new 'fd_type' and have a
> > custom protocol across this. Clients or input libraries would need to be
> > able to deal with this other protocol, which would be a big pain. For such
> > remote desktop use cases and even the Android one inputfd is not ideal.
>
> Admittedly, I hadn't thought of virtual/remote gaming devices but I think
> that it would be solvable over inputfd. Note, I'm just thinking out aloud
> here so the below has many holes awaiting some poking.
>
> Let's assume you have a pipe that you pass down via inputfd and you have a
> standardised text or binary protocol that you communicate over with. Then
> inputfd works in the intended manner, including easily closing the fd on
> focus out. you could even talk evdev over that pipe as long as the fd_type
> makes it clear it's a one-way communication (i.e. ioctls will fail). And in
> that case the client would have to get the device description from the
> proposed properties.
> So technically, it's certainly possible, whether it's *better* than a
> dedicated wayland protocol is quite debatable.
A pipe could work for input, but communication the other way is needed
as well for force feedback / rumble, which is quite common on game
controllers, joysticks and steering wheels. One way would be to use
two pipes, but the protocol would become more complex easily.
>
> > Now back to general inputfd feedback and requirements. I want to make sure
> > basic gamepad functionality is handled and also more complex functionality.
> > As the maintainer of hid-sony for ds3/ds4 controllers, I want to make sure
> > our gamepads are supported well. The same feedback would also apply to
> > xbox360, switch or steam controllers as all devices have become more
> > complex.
> >
> > The current proposal could work for exposing basic gamepad functionality
> > for the axes/buttons. Taking hid-sony as an example ds3 and ds4 devices are
> > now threated as composite devices utilizing multiple device nodes. The ds4
> > is probably the best example using 3 evdev nodes, 1 for gamepad, 1 for
> > touchpad (reported as a pointer device) and 1 for motion sensors. I expect
> > the Switch Pro controller and the Joypads for the Switch to use at minimum
> > 2 nodes as well.
> >
> > How would composite devices be handled? There would need to be a way to tie
> > multiple devices together. So far composite devices use all the same ids
> > (product/vendor) and same unique id (EVIOCGUNIQ) and even physical location
> > (EVIOCGPHYS). At minimum I think the compositor needs to tie these together
> > with some sort of shared id.
>
> what we do in libinput for these is a 'device group', which is basically an
> opaque identifier that tells you when two devices belong together. It's not
> specified *how* libinput gets to that and we have some device-specific hacks
> but it generally works fine. Here's a post from 2 years ago explaining that:
> http://who-t.blogspot.com.au/2015/02/libinput-device-groups.html
> That device group would be a good example for things we have to send down to
> the client as properties.
>
> I don't think that device group handling complicates things too much, the
> nodes are already divided so they need to be merged back together
> *somewhere*. Maybe in the mythical libgamingdev ;)
>
> > Another issue again taking ds4 as an example
> > it has a touchpad, which is already picked up as a pointer device
> > elsewhere. How to handle this? Unflag it as a pointer and share this
> > inputfd as well or use it really as a pointer and gamepad state the other
> > way?
>
> That's quite a hardware-specific question. But yeah, for this device and
> unless you need that touchpad in the normal desktop environment case, I
> would say you'd have to ship some configuration that takes ID_INPUT_TOUCHPAD
> off the touchpad. Coincidentally, I think this is also an argument to have
> our own JOYSTICK_TYPE property rather than overloading the ID_INPUT
> properties with more information.
>
> Note that (as pointed out in the other email 2 min ago) the compositor can
> use a device even when it forwards the inputfd to a client. It's all a
> matter of compositor-specific policy, but I don't see this as a dealbreaker
> yet.
>
> > In addition next to evdev nodes there are sysfs nodes for which fd passing
> > won't work. The best example would be LEDs as found on ds3/ds4/xbox360, but
> > I'm sure there will be GPIO as well. In case of ds4 it is not uncommon to
> > update the LEDs many times a second (often used for light effects). How
> > would such features be exposed? Is the idea to leverage properties or
> > something like this? Passing these nodes to clients is probably not a good
> > idea.
>
> tbh, I don't have a good answer to that but I'll take suggestions.
> This is a wayland protocol, so we can add requests for things that we can't
> handle over the fd directly - it's just a matter of how complex this gets
> before we end up regretting passing fds in the first place...
I don't have a good solution for the LED/gpio sysfs example I gave before.
Another sysfs example which I forgot about is power / battery related
information. This is typically done through sysfs as well and I think
the protocol needs to support this properly. This information might
even be something the compositor wants to know about like for showing
a warning about the battery level dropping. For gaming devices this is
a big concern as batteries often last less than a day as opposed to
wireless keyboards/mice which can even last a year.
>
> > This is some initial feedback. I'm not sure how much I like the inputfd
> > proposal. For a part because it doesn't fit some of our use cases well
> > (remote desktop + custom protocols). In addition I'm also a bit worried
> > about the complexity in handling composite devices.
>
> I'm under no illusion that this is more than an initial draft to get the
> discussion started, so feedback like yours is very important. Feel free to
> add bits to it so it solves your use-cases, eventually we'll get to
> something acceptable to everyone.
>
> Cheers,
> Peter
>
> >
> > Thanks,
> > Roderick
> >
> > --
> > Roderick Colenbrander
> > Senior Manager of Software Engineering
> > Gaikai, a Sony Interactive Entertainment Company
> > roderick at gaikai.com
> >
> >
> > On Fri, Mar 31, 2017 at 12:29 AM, Peter Hutterer <peter.hutterer at who-t.net>
> > wrote:
> >
> > > This is the first draft for a protocol to enable direct access to input
> > > devices. The basic premise is: instead of protocols to handle gaming
> > > devices, 3D mice, sensors, etc. we just hand you (the client) an fd and
> > > hope
> > > you're happy with it. You get to play with that fd until it's revoked, i.e.
> > > right now we only provide it for evdev fds where we have EVIOCMUTE.
> > >
> > > Focus management is straightforward and follows compositor policy. The
> > > protocol is quite simple:
> > >
> > > clients get a wp_inputfd_device for each device
> > > this device sends some static information (name, usb ids, ...)
> > > this device sends focus-in with an fd
> > > this device sends focus-out, the fd is revoked
> > > ... repeat focus in/out as necessary ...
> > > this device sends removed on unplug
> > >
> > > The devices are wrapped up into seats and those seats are provided by a
> > > global manager object - largely because that's what we need for the
> > > protocol
> > > to work.
> > >
> > > The notable bits are:
> > > * devices are divided into categories, so instead of getting all devices,
> > > you say "give me an inputfd-seat for gaming devices". That object then
> > > sends you all devices. In the future, we'll have an equivalent "give me
> > > the seat for 3D mice", etc., but right now the gaming devices is
> > > all that exists
> > > * there's a generic key/value event called "property". This one is largely
> > > so the compositor can tag devices with some capabilities to make it
> > > easier
> > > for the client to identify what it wants. I'll punt to Bastien for
> > > details because he requested that, but I can imagine something like
> > > "joystick type" is "gamepad". This big deal with this is that the
> > > dictionary of tags is something that compositor and clients will have to
> > > agree on, and it's not something defined by the protocol. This enables us
> > > to change/extend the tags independent of protocol bumps.
> > > It's quite similar to having access to the udev device to get at udev
> > > properties, but some devices may not have a udev device, so...
> > >
> > > What is *not* in the protocol is how the compositor decides what is a
> > > gaming
> > > devices. The ID_INPUT_JOYSTICK udev property is too crude with many false
> > > positives. For tablets, we get around this by having libwacom spit out udev
> > > rules to tag things correctly. A similar approach is the best solution
> > > here: have some database that knows about various joystick devices and
> > > which
> > > category they fall into and install the udev rules/hwdb entries
> > > accordingly. There was talk of a gaming-device-oriented library
> > > (libgamingdev), this library could provide that information. Such a library
> > > would also be a prime spot for that property dictionary. Volunteers to
> > > start
> > > working on this, please step forward!
> > >
> > > /me takes a big step back
> > >
> > > Again, this is a draft, so feel free to pick it apart or request changes so
> > > it matches your use-case. Please do add CC's as required, I am by no means
> > > a
> > > game developer so I'm not aware of use-cases beyond the immediately
> > > obvious.
> > >
> > > Cheers,
> > > Peter
> > >
> > > ---
> > >
> > > diff --git a/Makefile.am b/Makefile.am
> > > index e693afa..e46910a 100644
> > > --- a/Makefile.am
> > > +++ b/Makefile.am
> > > @@ -4,6 +4,7 @@ unstable_protocols =
> > > \
> > > unstable/linux-dmabuf/linux-dmabuf-unstable-v1.xml
> > > \
> > > unstable/text-input/text-input-unstable-v1.xml
> > > \
> > > unstable/input-method/input-method-unstable-v1.xml
> > > \
> > > + unstable/inputfd/inputfd-unstable-v1.xml
> > > \
> > > unstable/xdg-shell/xdg-shell-unstable-v5.xml
> > > \
> > > unstable/xdg-shell/xdg-shell-unstable-v6.xml
> > > \
> > > unstable/relative-pointer/relative-pointer-unstable-v1.xml
> > > \
> > > diff --git a/unstable/inputfd/README b/unstable/inputfd/README
> > > new file mode 100644
> > > index 0000000..a24d858
> > > --- /dev/null
> > > +++ b/unstable/inputfd/README
> > > @@ -0,0 +1,4 @@
> > > +Input device fd passing protocol
> > > +
> > > +Maintainers:
> > > +Peter Hutterer <peter.hutterer at who-t.net>
> > > diff --git a/unstable/inputfd/inputfd-unstable-v1.xml
> > > b/unstable/inputfd/inputfd-unstable-v1.xml
> > > new file mode 100644
> > > index 0000000..3b83378
> > > --- /dev/null
> > > +++ b/unstable/inputfd/inputfd-unstable-v1.xml
> > > @@ -0,0 +1,279 @@
> > > +<?xml version="1.0" encoding="UTF-8"?>
> > > +<protocol name="inputfd_unstable_v1">
> > > + <copyright>
> > > + Copyright 2017 © Red Hat, Inc.
> > > +
> > > + Permission is hereby granted, free of charge, to any person
> > > + obtaining a copy of this software and associated documentation files
> > > + (the "Software"), to deal in the Software without restriction,
> > > + including without limitation the rights to use, copy, modify, merge,
> > > + publish, distribute, sublicense, and/or sell copies of the Software,
> > > + and to permit persons to whom the Software is furnished to do so,
> > > + subject to the following conditions:
> > > +
> > > + The above copyright notice and this permission notice (including the
> > > + next paragraph) shall be included in all copies or substantial
> > > + portions of the Software.
> > > +
> > > + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> > > + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> > > + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> > > + NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
> > > + BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> > > + ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> > > + CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> > > + SOFTWARE.
> > > + </copyright>
> > > +
> > > + <description summary="Wayland protocol for direct fd access to input
> > > devices">
> > > + This description provides a high-level overview of the interfaces
> > > + in this protocol. For details, see the protocol specification.
> > > +
> > > + Some input devices do not interact with the windowing system. Examples
> > > + of such input devices are gaming controllers or 3D mice. In many
> > > cases,
> > > + a client requires direct access to the device to access or interpret
> > > + device-specific functionality.
> > > +
> > > + This interface provides the ability for a compositor to pass a file
> > > + descriptor to the client. The compositor may restrict the type of
> > > device
> > > + designated as compatible device and it may restrict specific events
> > > from
> > > + being sent to the client (e.g. by masking the Home button on a
> > > gamepad).
> > > + Otherwise, a client should treat the device as if opened manually.
> > > +
> > > + This interface divides devices into type-based categories, clients are
> > > + expected to only request access to devices that match the category
> > > they
> > > + require. As of version 1, this interface provides only a single
> > > + category: "gaming devices". Future categories of devices are expected
> > > to
> > > + re-use the same wp_inputfd_device interface, but may differ in the way
> > > + how to request them.
> > > +
> > > + Multiple input devices of the same type may exists and assigned to
> > > + different seats. The top-level object of this protocol is a
> > > + wp_inputfd_manager. Dependent on the device type sought after, a
> > > client
> > > + may request a type-specific seat interface (e.g.
> > > + wp_inputfd_manager.get_seat_gamedev). This seat then provides the
> > > list
> > > + of devices for that category.
> > > +
> > > + Once a compositor deems a device to be focused on the client or on a
> > > + client's surface it sends a wp_inputfd_device.focus_in event with a
> > > file
> > > + descriptor for this device. A compositor may arbitrarily revoke access
> > > + to the device by sending a wp_inputfd_gamedev.focus_out.
> > > Additionally, a
> > > + compositor may invoke system functionality to restrict access to the
> > > + file descriptor, e.g. by using EVIOCMUTE on an evdev fd.
> > > +
> > > + Otherwise, a client should treat the file descriptor as direct access
> > > to
> > > + the device for the duration of it having access.
> > > + </description>
> > > +
> > > + <interface name="zwp_inputfd_manager_v1" version="1">
> > > + <description summary="controller object for direct fd access input
> > > devices">
> > > + An object that provides access to the input devices available for
> > > + direct fd access on this system. All input devices are associated
> > > with
> > > + a seat, to get access to the actual devices, use one of the get_seat
> > > + requests, e.g. wp_inputfd_manager.get_seat_gamedev.
> > > + </description>
> > > +
> > > + <request name="get_seat_gamedev">
> > > + <description summary="get the seat for gaming devices">
> > > + Get the wp_inputfd_seat object for the given seat. This object
> > > + provides access to all gaming devices in this seat.
> > > +
> > > + The decision which device is a gaming device is made by the
> > > + compositor. The protocol makes no guarantees whether a particular
> > > + device is available through this interface.
> > > + </description>
> > > + <arg name="inputfd_seat" type="new_id" interface="zwp_inputfd_seat_
> > > v1"/>
> > > + <arg name="seat" type="object" interface="wl_seat" summary="The
> > > wl_seat object to retrieve the input devices for" />
> > > + </request>
> > > +
> > > + <request name="destroy" type="destructor">
> > > + <description summary="release the memory for the inputfd manager
> > > object">
> > > + Destroy the wp_inputfd_manager object. Objects created from this
> > > + object are unaffected and should be destroyed separately.
> > > + </description>
> > > + </request>
> > > + </interface>
> > > +
> > > + <interface name="zwp_inputfd_seat_v1" version="1">
> > > + <description summary="controller object for input devices of a seat">
> > > + An object that provides access to the input devices available on
> > > this
> > > + seat for the requested type of device. After binding to this
> > > + interface, the compositor sends a set of
> > > wp_inputfd_seat.device_added
> > > + events for currently available devices and whenever a new device
> > > + becomes available.
> > > + </description>
> > > +
> > > + <request name="destroy" type="destructor">
> > > + <description summary="release the memory for the inputfd seat
> > > object">
> > > + Destroy the wp_inputfd_seat object. Objects created from this
> > > + object are unaffected and should be destroyed separately.
> > > + </description>
> > > + </request>
> > > +
> > > + <event name="device_added">
> > > + <description summary="new device notification">
> > > + This event is sent whenever a new device becomes available on
> > > + this seat. This event only provides the object id of the devices,
> > > + any static information about the device (device name,
> > > + vid/pid, etc.) is sent through the wp_inputfd_device interface.
> > > +
> > > + Which devices are compatible input devices for this seat is a
> > > + decision made by the compositor, the protocol makes no guarantee
> > > + that any specific device becomes available as inputfd device to a
> > > + client.
> > > + </description>
> > > + <arg name="id" type="new_id" interface="zwp_inputfd_device_v1"
> > > summary="the newly added device"/>
> > > + </event>
> > > + </interface>
> > > +
> > > + <interface name="zwp_inputfd_device_v1" version="1">
> > > + <description summary="input fd device">
> > > + The wp_inputfd_device interface represents one device accessible
> > > + directly by an fd passed to the client.
> > > +
> > > + A device has a number of static characteristics, e.g. device
> > > + name and pid/vid. These capabilities are sent in an event sequence
> > > + immediately after the wp_inputfd_seat.device_added event. This
> > > initial
> > > + event sequence is terminated by a wp_inputfd_device.done event. This
> > > + sequence is sent only once and always before the first
> > > + wp_inputfd_device.focus_in event.
> > > + </description>
> > > +
> > > + <request name="destroy" type="destructor">
> > > + <description summary="destroy the inputfd object">
> > > + This destroys the client's resource for this inputfd object.
> > > + </description>
> > > + </request>
> > > +
> > > + <event name="name">
> > > + <description summary="device name">
> > > + The name is a UTF-8 encoded string with the device's name, intended
> > > + for presentation to the user.
> > > +
> > > + This event is sent in the initial burst of events before the
> > > + wp_inputfd_device.done event.
> > > +
> > > + This event is optional, if the required information is not
> > > available
> > > + for this device the event is omitted.
> > > + </description>
> > > + <arg name="name" type="string" summary="the device name"/>
> > > + </event>
> > > +
> > > + <event name="usb_id">
> > > + <description summary="device USB vendor/product id">
> > > + This event is sent in the initial burst of events before the
> > > + wp_inputfd_device.done event.
> > > +
> > > + This event is optional, if the required information is not
> > > available
> > > + for this device the event is omitted.
> > > + </description>
> > > + <arg name="vid" type="uint" summary="USB vendor id"/>
> > > + <arg name="pid" type="uint" summary="USB product id"/>
> > > + </event>
> > > +
> > > + <event name="property">
> > > + <description summary="device capability notification">
> > > + This event is sent to notify the client of a custom property that
> > > + applies to this device. The property is a standard key/value store
> > > + in UTF-8 format, interpretation of both strings is left to the
> > > + client. The wayland protocol makes no guarantees about the content
> > > + of each string beyond its text encoding.
> > > +
> > > + Compositors and clients need to agree on a dictionary of
> > > properties.
> > > + For example, a compositor may designate the device to be of
> > > + 'joystick-type' 'gamepad'. This dictionary is out of the scope of
> > > + this protocol.
> > > + </description>
> > > + <arg name="property" type="string" summary="A UTF-8 encoded
> > > property name"/>
> > > + <arg name="value" type="string" summary="A UTF-8 encoded property
> > > value"/>
> > > + </event>
> > > +
> > > + <event name="done">
> > > + <description summary="device description events sequence complete">
> > > + This event is sent immediately to signal the end of the initial
> > > + burst of descriptive events. A client may consider the static
> > > + description of the device to be complete and finalize
> > > + initialization of the device.
> > > + </description>
> > > + </event>
> > > +
> > > + <event name="removed">
> > > + <description summary="device removed event">
> > > + Sent when the device has been removed from the system.
> > > +
> > > + If the client currently has the device focus, a
> > > + wp_inputfd_device.focus_out event is sent before the removed event.
> > > + See wp_inputfd_device.focus_in for more details.
> > > +
> > > + When this event is received, the client must
> > > wp_inputfd_device.destroy
> > > + the object.
> > > + </description>
> > > + </event>
> > > +
> > > + <enum name="fd_type">
> > > + <description summary="Input fd device file descriptor types">
> > > + This enum specifies the format of the file descriptor passed to
> > > + clients with the wp_inputfd_device.focus_in event.
> > > + </description>
> > > + <entry name="evdev" value="0" summary="An evdev file descriptor" />
> > > + </enum>
> > > +
> > > + <event name="focus_in">
> > > + <description summary="input fd device focus in event">
> > > + Notification that this client now has the focus and/or access to
> > > + this device. The decision what consitutes focus left to the
> > > + compositor. For example, a compositor may tie joystick focus to the
> > > + wl_pointer focus of this seat. The protocol does not guarantee that
> > > + any specific client ever receives the focus for a device.
> > > +
> > > + The client is passed a file descriptor with access to this
> > > + device. This file descriptor is valid until a subsequent
> > > + wp_inputfd_device.focus_out event. Upon
> > > wp_inputfd_device.focus_out, the
> > > + compositor may revoke the fd and further operations will fail.
> > > +
> > > + However, due to potential race conditions a client must be able to
> > > + handle errors as if it opened the fd itself. No guarantee is
> > > + given that the wp_inputfd_device.focus_out event or
> > > wp_inputfd_device.removed
> > > + event are sent before the client encounters an error on the file
> > > + descriptor.
> > > +
> > > + A compositor guarantees that the underlying device does not change
> > > + until a wp_inputfd_device.removed event. In other words, if the fd
> > > + type allows querying capabilities through the fd, a client needs to
> > > + do so only once at the first focus_in. Subsequent focus_in events
> > > + will provide the same capabilities.
> > > +
> > > + If applicable, this event contains the surface that has the focus..
> > > + In some cases, the focus may not be tied to a specific client
> > > surface
> > > + but is given to the client independent of any surface. In that
> > > case,
> > > + the surface is null.
> > > +
> > > + The protocol guarantees that focus_in and focus_out always come in
> > > + pairs. If the client currently has the focus and the device is
> > > + removed, a focus_out event is sent to the client before the
> > > + wp_inputfd_device.removed event.
> > > + </description>
> > > + <arg name="serial" type="uint"/>
> > > + <arg name="fd" type="fd" summary="file descriptor to the device"/>
> > > + <arg name="fd_type" type="uint" enum="fd_type" summary="fd type" />
> > > + <arg name="surface" type="object" interface="wl_surface"
> > > summary="The current surface that has the device's focus"
> > > allow-null="true"/>
> > > + </event>
> > > +
> > > + <event name="focus_out">
> > > + <description summary="input fd device focus out event">
> > > + Notification that this client no longer has focus and/or access to
> > > + this device. Further reads from this device's file descriptor
> > > + will fail. The client must close(2) the file descriptor received in
> > > + the wp_inputfd_device.focus_in event.
> > > +
> > > + This event does not mean the device was removed, merely that the
> > > + device is focused elsewhere. For device removal, see
> > > + wp_inputfd_device.removed.
> > > +
> > > + See wp_inputfd_device.focus_in for more details.
> > > + </description>
> > > + </event>
> > > + </interface>
> > > +</protocol>
> > > +
> > > +
> > >
More information about the wayland-devel
mailing list