[RFC wayland-protocols] inputfd - direct input access protocol
Peter Hutterer
peter.hutterer at who-t.net
Tue Apr 4 04:08:45 UTC 2017
On Mon, Apr 03, 2017 at 04:08:21PM -0700, Roderick Colenbrander wrote:
> Hi Peter,
>
> Thanks for sharing this proposal. I have some little comments for a later
> stage, but would rather discuss some big items first.
>
> The feedback I will share comes from a couple different angles as we are
> interested in Wayland for various different use cases.
good :) that's exactly what we need right now to get this right.
> Originally we had the gamepad wayland thread originally started by Jingkui.
> The proposal there was to provide knowledge of axes, buttons and everything
> making up a gamepad to the compositor. He proposed this direction as a way
> to inject events from Chrome into Android as at that level there is no file
> descriptor anymore. Ourselves we were quite interested in this approach as
> well. Among the things we do is roughly remote streamable desktops.
> Currently on X you can inject keyboard/mouse events through XTest, but the
> same applies to a custom Wayland compositor into which you can easily
> inject keyboard/mouse data. Clients don't have to be aware of the custom
> virtual devices and protocols behind the scenes.
>
> When using inputfd for such a use case, either we would need to fake an
> input device through uinput (not ideal) or add a new 'fd_type' and have a
> custom protocol across this. Clients or input libraries would need to be
> able to deal with this other protocol, which would be a big pain. For such
> remote desktop use cases and even the Android one inputfd is not ideal.
Admittedly, I hadn't thought of virtual/remote gaming devices but I think
that it would be solvable over inputfd. Note, I'm just thinking out aloud
here so the below has many holes awaiting some poking.
Let's assume you have a pipe that you pass down via inputfd and you have a
standardised text or binary protocol that you communicate over with. Then
inputfd works in the intended manner, including easily closing the fd on
focus out. you could even talk evdev over that pipe as long as the fd_type
makes it clear it's a one-way communication (i.e. ioctls will fail). And in
that case the client would have to get the device description from the
proposed properties.
So technically, it's certainly possible, whether it's *better* than a
dedicated wayland protocol is quite debatable.
> Now back to general inputfd feedback and requirements. I want to make sure
> basic gamepad functionality is handled and also more complex functionality.
> As the maintainer of hid-sony for ds3/ds4 controllers, I want to make sure
> our gamepads are supported well. The same feedback would also apply to
> xbox360, switch or steam controllers as all devices have become more
> complex.
>
> The current proposal could work for exposing basic gamepad functionality
> for the axes/buttons. Taking hid-sony as an example ds3 and ds4 devices are
> now threated as composite devices utilizing multiple device nodes. The ds4
> is probably the best example using 3 evdev nodes, 1 for gamepad, 1 for
> touchpad (reported as a pointer device) and 1 for motion sensors. I expect
> the Switch Pro controller and the Joypads for the Switch to use at minimum
> 2 nodes as well.
>
> How would composite devices be handled? There would need to be a way to tie
> multiple devices together. So far composite devices use all the same ids
> (product/vendor) and same unique id (EVIOCGUNIQ) and even physical location
> (EVIOCGPHYS). At minimum I think the compositor needs to tie these together
> with some sort of shared id.
what we do in libinput for these is a 'device group', which is basically an
opaque identifier that tells you when two devices belong together. It's not
specified *how* libinput gets to that and we have some device-specific hacks
but it generally works fine. Here's a post from 2 years ago explaining that:
http://who-t.blogspot.com.au/2015/02/libinput-device-groups.html
That device group would be a good example for things we have to send down to
the client as properties.
I don't think that device group handling complicates things too much, the
nodes are already divided so they need to be merged back together
*somewhere*. Maybe in the mythical libgamingdev ;)
> Another issue again taking ds4 as an example
> it has a touchpad, which is already picked up as a pointer device
> elsewhere. How to handle this? Unflag it as a pointer and share this
> inputfd as well or use it really as a pointer and gamepad state the other
> way?
That's quite a hardware-specific question. But yeah, for this device and
unless you need that touchpad in the normal desktop environment case, I
would say you'd have to ship some configuration that takes ID_INPUT_TOUCHPAD
off the touchpad. Coincidentally, I think this is also an argument to have
our own JOYSTICK_TYPE property rather than overloading the ID_INPUT
properties with more information.
Note that (as pointed out in the other email 2 min ago) the compositor can
use a device even when it forwards the inputfd to a client. It's all a
matter of compositor-specific policy, but I don't see this as a dealbreaker
yet.
> In addition next to evdev nodes there are sysfs nodes for which fd passing
> won't work. The best example would be LEDs as found on ds3/ds4/xbox360, but
> I'm sure there will be GPIO as well. In case of ds4 it is not uncommon to
> update the LEDs many times a second (often used for light effects). How
> would such features be exposed? Is the idea to leverage properties or
> something like this? Passing these nodes to clients is probably not a good
> idea.
tbh, I don't have a good answer to that but I'll take suggestions.
This is a wayland protocol, so we can add requests for things that we can't
handle over the fd directly - it's just a matter of how complex this gets
before we end up regretting passing fds in the first place...
> This is some initial feedback. I'm not sure how much I like the inputfd
> proposal. For a part because it doesn't fit some of our use cases well
> (remote desktop + custom protocols). In addition I'm also a bit worried
> about the complexity in handling composite devices.
I'm under no illusion that this is more than an initial draft to get the
discussion started, so feedback like yours is very important. Feel free to
add bits to it so it solves your use-cases, eventually we'll get to
something acceptable to everyone.
Cheers,
Peter
>
> Thanks,
> Roderick
>
> --
> Roderick Colenbrander
> Senior Manager of Software Engineering
> Gaikai, a Sony Interactive Entertainment Company
> roderick at gaikai.com
>
>
> On Fri, Mar 31, 2017 at 12:29 AM, Peter Hutterer <peter.hutterer at who-t.net>
> wrote:
>
> > This is the first draft for a protocol to enable direct access to input
> > devices. The basic premise is: instead of protocols to handle gaming
> > devices, 3D mice, sensors, etc. we just hand you (the client) an fd and
> > hope
> > you're happy with it. You get to play with that fd until it's revoked, i.e.
> > right now we only provide it for evdev fds where we have EVIOCMUTE.
> >
> > Focus management is straightforward and follows compositor policy. The
> > protocol is quite simple:
> >
> > clients get a wp_inputfd_device for each device
> > this device sends some static information (name, usb ids, ...)
> > this device sends focus-in with an fd
> > this device sends focus-out, the fd is revoked
> > ... repeat focus in/out as necessary ...
> > this device sends removed on unplug
> >
> > The devices are wrapped up into seats and those seats are provided by a
> > global manager object - largely because that's what we need for the
> > protocol
> > to work.
> >
> > The notable bits are:
> > * devices are divided into categories, so instead of getting all devices,
> > you say "give me an inputfd-seat for gaming devices". That object then
> > sends you all devices. In the future, we'll have an equivalent "give me
> > the seat for 3D mice", etc., but right now the gaming devices is
> > all that exists
> > * there's a generic key/value event called "property". This one is largely
> > so the compositor can tag devices with some capabilities to make it
> > easier
> > for the client to identify what it wants. I'll punt to Bastien for
> > details because he requested that, but I can imagine something like
> > "joystick type" is "gamepad". This big deal with this is that the
> > dictionary of tags is something that compositor and clients will have to
> > agree on, and it's not something defined by the protocol. This enables us
> > to change/extend the tags independent of protocol bumps.
> > It's quite similar to having access to the udev device to get at udev
> > properties, but some devices may not have a udev device, so...
> >
> > What is *not* in the protocol is how the compositor decides what is a
> > gaming
> > devices. The ID_INPUT_JOYSTICK udev property is too crude with many false
> > positives. For tablets, we get around this by having libwacom spit out udev
> > rules to tag things correctly. A similar approach is the best solution
> > here: have some database that knows about various joystick devices and
> > which
> > category they fall into and install the udev rules/hwdb entries
> > accordingly. There was talk of a gaming-device-oriented library
> > (libgamingdev), this library could provide that information. Such a library
> > would also be a prime spot for that property dictionary. Volunteers to
> > start
> > working on this, please step forward!
> >
> > /me takes a big step back
> >
> > Again, this is a draft, so feel free to pick it apart or request changes so
> > it matches your use-case. Please do add CC's as required, I am by no means
> > a
> > game developer so I'm not aware of use-cases beyond the immediately
> > obvious.
> >
> > Cheers,
> > Peter
> >
> > ---
> >
> > diff --git a/Makefile.am b/Makefile.am
> > index e693afa..e46910a 100644
> > --- a/Makefile.am
> > +++ b/Makefile.am
> > @@ -4,6 +4,7 @@ unstable_protocols =
> > \
> > unstable/linux-dmabuf/linux-dmabuf-unstable-v1.xml
> > \
> > unstable/text-input/text-input-unstable-v1.xml
> > \
> > unstable/input-method/input-method-unstable-v1.xml
> > \
> > + unstable/inputfd/inputfd-unstable-v1.xml
> > \
> > unstable/xdg-shell/xdg-shell-unstable-v5.xml
> > \
> > unstable/xdg-shell/xdg-shell-unstable-v6.xml
> > \
> > unstable/relative-pointer/relative-pointer-unstable-v1.xml
> > \
> > diff --git a/unstable/inputfd/README b/unstable/inputfd/README
> > new file mode 100644
> > index 0000000..a24d858
> > --- /dev/null
> > +++ b/unstable/inputfd/README
> > @@ -0,0 +1,4 @@
> > +Input device fd passing protocol
> > +
> > +Maintainers:
> > +Peter Hutterer <peter.hutterer at who-t.net>
> > diff --git a/unstable/inputfd/inputfd-unstable-v1.xml
> > b/unstable/inputfd/inputfd-unstable-v1.xml
> > new file mode 100644
> > index 0000000..3b83378
> > --- /dev/null
> > +++ b/unstable/inputfd/inputfd-unstable-v1.xml
> > @@ -0,0 +1,279 @@
> > +<?xml version="1.0" encoding="UTF-8"?>
> > +<protocol name="inputfd_unstable_v1">
> > + <copyright>
> > + Copyright 2017 © Red Hat, Inc.
> > +
> > + Permission is hereby granted, free of charge, to any person
> > + obtaining a copy of this software and associated documentation files
> > + (the "Software"), to deal in the Software without restriction,
> > + including without limitation the rights to use, copy, modify, merge,
> > + publish, distribute, sublicense, and/or sell copies of the Software,
> > + and to permit persons to whom the Software is furnished to do so,
> > + subject to the following conditions:
> > +
> > + The above copyright notice and this permission notice (including the
> > + next paragraph) shall be included in all copies or substantial
> > + portions of the Software.
> > +
> > + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> > + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> > + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> > + NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
> > + BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> > + ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> > + CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> > + SOFTWARE.
> > + </copyright>
> > +
> > + <description summary="Wayland protocol for direct fd access to input
> > devices">
> > + This description provides a high-level overview of the interfaces
> > + in this protocol. For details, see the protocol specification.
> > +
> > + Some input devices do not interact with the windowing system. Examples
> > + of such input devices are gaming controllers or 3D mice. In many
> > cases,
> > + a client requires direct access to the device to access or interpret
> > + device-specific functionality.
> > +
> > + This interface provides the ability for a compositor to pass a file
> > + descriptor to the client. The compositor may restrict the type of
> > device
> > + designated as compatible device and it may restrict specific events
> > from
> > + being sent to the client (e.g. by masking the Home button on a
> > gamepad).
> > + Otherwise, a client should treat the device as if opened manually.
> > +
> > + This interface divides devices into type-based categories, clients are
> > + expected to only request access to devices that match the category
> > they
> > + require. As of version 1, this interface provides only a single
> > + category: "gaming devices". Future categories of devices are expected
> > to
> > + re-use the same wp_inputfd_device interface, but may differ in the way
> > + how to request them.
> > +
> > + Multiple input devices of the same type may exists and assigned to
> > + different seats. The top-level object of this protocol is a
> > + wp_inputfd_manager. Dependent on the device type sought after, a
> > client
> > + may request a type-specific seat interface (e.g.
> > + wp_inputfd_manager.get_seat_gamedev). This seat then provides the
> > list
> > + of devices for that category.
> > +
> > + Once a compositor deems a device to be focused on the client or on a
> > + client's surface it sends a wp_inputfd_device.focus_in event with a
> > file
> > + descriptor for this device. A compositor may arbitrarily revoke access
> > + to the device by sending a wp_inputfd_gamedev.focus_out.
> > Additionally, a
> > + compositor may invoke system functionality to restrict access to the
> > + file descriptor, e.g. by using EVIOCMUTE on an evdev fd.
> > +
> > + Otherwise, a client should treat the file descriptor as direct access
> > to
> > + the device for the duration of it having access.
> > + </description>
> > +
> > + <interface name="zwp_inputfd_manager_v1" version="1">
> > + <description summary="controller object for direct fd access input
> > devices">
> > + An object that provides access to the input devices available for
> > + direct fd access on this system. All input devices are associated
> > with
> > + a seat, to get access to the actual devices, use one of the get_seat
> > + requests, e.g. wp_inputfd_manager.get_seat_gamedev.
> > + </description>
> > +
> > + <request name="get_seat_gamedev">
> > + <description summary="get the seat for gaming devices">
> > + Get the wp_inputfd_seat object for the given seat. This object
> > + provides access to all gaming devices in this seat.
> > +
> > + The decision which device is a gaming device is made by the
> > + compositor. The protocol makes no guarantees whether a particular
> > + device is available through this interface.
> > + </description>
> > + <arg name="inputfd_seat" type="new_id" interface="zwp_inputfd_seat_
> > v1"/>
> > + <arg name="seat" type="object" interface="wl_seat" summary="The
> > wl_seat object to retrieve the input devices for" />
> > + </request>
> > +
> > + <request name="destroy" type="destructor">
> > + <description summary="release the memory for the inputfd manager
> > object">
> > + Destroy the wp_inputfd_manager object. Objects created from this
> > + object are unaffected and should be destroyed separately.
> > + </description>
> > + </request>
> > + </interface>
> > +
> > + <interface name="zwp_inputfd_seat_v1" version="1">
> > + <description summary="controller object for input devices of a seat">
> > + An object that provides access to the input devices available on
> > this
> > + seat for the requested type of device. After binding to this
> > + interface, the compositor sends a set of
> > wp_inputfd_seat.device_added
> > + events for currently available devices and whenever a new device
> > + becomes available.
> > + </description>
> > +
> > + <request name="destroy" type="destructor">
> > + <description summary="release the memory for the inputfd seat
> > object">
> > + Destroy the wp_inputfd_seat object. Objects created from this
> > + object are unaffected and should be destroyed separately.
> > + </description>
> > + </request>
> > +
> > + <event name="device_added">
> > + <description summary="new device notification">
> > + This event is sent whenever a new device becomes available on
> > + this seat. This event only provides the object id of the devices,
> > + any static information about the device (device name,
> > + vid/pid, etc.) is sent through the wp_inputfd_device interface.
> > +
> > + Which devices are compatible input devices for this seat is a
> > + decision made by the compositor, the protocol makes no guarantee
> > + that any specific device becomes available as inputfd device to a
> > + client.
> > + </description>
> > + <arg name="id" type="new_id" interface="zwp_inputfd_device_v1"
> > summary="the newly added device"/>
> > + </event>
> > + </interface>
> > +
> > + <interface name="zwp_inputfd_device_v1" version="1">
> > + <description summary="input fd device">
> > + The wp_inputfd_device interface represents one device accessible
> > + directly by an fd passed to the client.
> > +
> > + A device has a number of static characteristics, e.g. device
> > + name and pid/vid. These capabilities are sent in an event sequence
> > + immediately after the wp_inputfd_seat.device_added event. This
> > initial
> > + event sequence is terminated by a wp_inputfd_device.done event. This
> > + sequence is sent only once and always before the first
> > + wp_inputfd_device.focus_in event.
> > + </description>
> > +
> > + <request name="destroy" type="destructor">
> > + <description summary="destroy the inputfd object">
> > + This destroys the client's resource for this inputfd object.
> > + </description>
> > + </request>
> > +
> > + <event name="name">
> > + <description summary="device name">
> > + The name is a UTF-8 encoded string with the device's name, intended
> > + for presentation to the user.
> > +
> > + This event is sent in the initial burst of events before the
> > + wp_inputfd_device.done event.
> > +
> > + This event is optional, if the required information is not
> > available
> > + for this device the event is omitted.
> > + </description>
> > + <arg name="name" type="string" summary="the device name"/>
> > + </event>
> > +
> > + <event name="usb_id">
> > + <description summary="device USB vendor/product id">
> > + This event is sent in the initial burst of events before the
> > + wp_inputfd_device.done event.
> > +
> > + This event is optional, if the required information is not
> > available
> > + for this device the event is omitted.
> > + </description>
> > + <arg name="vid" type="uint" summary="USB vendor id"/>
> > + <arg name="pid" type="uint" summary="USB product id"/>
> > + </event>
> > +
> > + <event name="property">
> > + <description summary="device capability notification">
> > + This event is sent to notify the client of a custom property that
> > + applies to this device. The property is a standard key/value store
> > + in UTF-8 format, interpretation of both strings is left to the
> > + client. The wayland protocol makes no guarantees about the content
> > + of each string beyond its text encoding.
> > +
> > + Compositors and clients need to agree on a dictionary of
> > properties.
> > + For example, a compositor may designate the device to be of
> > + 'joystick-type' 'gamepad'. This dictionary is out of the scope of
> > + this protocol.
> > + </description>
> > + <arg name="property" type="string" summary="A UTF-8 encoded
> > property name"/>
> > + <arg name="value" type="string" summary="A UTF-8 encoded property
> > value"/>
> > + </event>
> > +
> > + <event name="done">
> > + <description summary="device description events sequence complete">
> > + This event is sent immediately to signal the end of the initial
> > + burst of descriptive events. A client may consider the static
> > + description of the device to be complete and finalize
> > + initialization of the device.
> > + </description>
> > + </event>
> > +
> > + <event name="removed">
> > + <description summary="device removed event">
> > + Sent when the device has been removed from the system.
> > +
> > + If the client currently has the device focus, a
> > + wp_inputfd_device.focus_out event is sent before the removed event.
> > + See wp_inputfd_device.focus_in for more details.
> > +
> > + When this event is received, the client must
> > wp_inputfd_device.destroy
> > + the object.
> > + </description>
> > + </event>
> > +
> > + <enum name="fd_type">
> > + <description summary="Input fd device file descriptor types">
> > + This enum specifies the format of the file descriptor passed to
> > + clients with the wp_inputfd_device.focus_in event.
> > + </description>
> > + <entry name="evdev" value="0" summary="An evdev file descriptor" />
> > + </enum>
> > +
> > + <event name="focus_in">
> > + <description summary="input fd device focus in event">
> > + Notification that this client now has the focus and/or access to
> > + this device. The decision what consitutes focus left to the
> > + compositor. For example, a compositor may tie joystick focus to the
> > + wl_pointer focus of this seat. The protocol does not guarantee that
> > + any specific client ever receives the focus for a device.
> > +
> > + The client is passed a file descriptor with access to this
> > + device. This file descriptor is valid until a subsequent
> > + wp_inputfd_device.focus_out event. Upon
> > wp_inputfd_device.focus_out, the
> > + compositor may revoke the fd and further operations will fail.
> > +
> > + However, due to potential race conditions a client must be able to
> > + handle errors as if it opened the fd itself. No guarantee is
> > + given that the wp_inputfd_device.focus_out event or
> > wp_inputfd_device.removed
> > + event are sent before the client encounters an error on the file
> > + descriptor.
> > +
> > + A compositor guarantees that the underlying device does not change
> > + until a wp_inputfd_device.removed event. In other words, if the fd
> > + type allows querying capabilities through the fd, a client needs to
> > + do so only once at the first focus_in. Subsequent focus_in events
> > + will provide the same capabilities.
> > +
> > + If applicable, this event contains the surface that has the focus..
> > + In some cases, the focus may not be tied to a specific client
> > surface
> > + but is given to the client independent of any surface. In that
> > case,
> > + the surface is null.
> > +
> > + The protocol guarantees that focus_in and focus_out always come in
> > + pairs. If the client currently has the focus and the device is
> > + removed, a focus_out event is sent to the client before the
> > + wp_inputfd_device.removed event.
> > + </description>
> > + <arg name="serial" type="uint"/>
> > + <arg name="fd" type="fd" summary="file descriptor to the device"/>
> > + <arg name="fd_type" type="uint" enum="fd_type" summary="fd type" />
> > + <arg name="surface" type="object" interface="wl_surface"
> > summary="The current surface that has the device's focus"
> > allow-null="true"/>
> > + </event>
> > +
> > + <event name="focus_out">
> > + <description summary="input fd device focus out event">
> > + Notification that this client no longer has focus and/or access to
> > + this device. Further reads from this device's file descriptor
> > + will fail. The client must close(2) the file descriptor received in
> > + the wp_inputfd_device.focus_in event.
> > +
> > + This event does not mean the device was removed, merely that the
> > + device is focused elsewhere. For device removal, see
> > + wp_inputfd_device.removed.
> > +
> > + See wp_inputfd_device.focus_in for more details.
> > + </description>
> > + </event>
> > + </interface>
> > +</protocol>
> > +
> > +
> >
More information about the wayland-devel
mailing list