[RFC] libinputmapper: Input device configuration for graphic-servers

[David Herrmann]

Hi Peter

On Tue, May 21, 2013 at 6:37 AM, Peter Hutterer
<peter.hutterer at who-t.net> wrote:
> On Thu, May 16, 2013 at 03:16:11PM +0200, David Herrmann wrote:
>> Hi Peter
>>
>> On Thu, May 16, 2013 at 7:37 AM, Peter Hutterer
>> <peter.hutterer at who-t.net> wrote:
>> > On Sun, May 12, 2013 at 04:20:59PM +0200, David Herrmann wrote:
>> [..]
>> >> So what is the proposed solution?
>> >> My recommendation is, that compositors still search for devices via
>> >> udev and use device drivers like libxkbcommon. So linux evdev handling
>> >> is still controlled by the compositor. However, I'd like to see
>> >> something like my libinputmapper proposal being used for device
>> >> detection and classification.
>> >>
>> >> libinputmapper provides an "inmap_evdev" object which reads device
>> >> information from an evdev-fd or sysfs /sys/class/input/input<num>
>> >> path, performs some heuristics to classify it and searches it's global
>> >> database for known fixups for broken devices.
>> >> It then provides "capabilities" to the caller, which allow them to see
>> >> what drivers to load on the device. And it provides a very simple
>> >> mapping table that allows to apply fixup mappings for broken devices.
>> >> These mappings are simple 1-to-1 mappings that are supposed to be
>> >> applied before drivers handle the input. This is to avoid
>> >> device-specific fixup in the drivers and move all this to the
>> >> inputmapper. An example would be a remapping for gamepads that report
>> >> BTN_A instead of BTN_NORTH, but we cannot fix them in the kernel for
>> >> backwards-compatibility reasons. The gamepad-driver can then assume
>> >> that if it receives BTN_NORTH, it is guaranteed to be BTN_NORTH and
>> >> doesn't need to special case xbox360/etc. controllers, because they're
>> >> broken.
>> >
>> > I think evdev is exactly that interface and apparently it doesn't work.
>> >
>> > if you want a mapping table, you need a per-client table because sooner or
>> > later you have a client that needs BTN_FOO when the kernel gives you BTN_BAR
>> > and you can't change the client to fix it.
>> >
>> > i.e. the same issue evdev has now, having a global remapping table just
>> > moves the problem down by 2 years.
>> >
>> > a mapping table is good, but you probably want two stages of mapping: one
>> > that's used in the compositor for truly broken devices that for some reason
>> > can't be fixed in the kernel, and one that's used on a per-client basis. and
>> > you'll likely want to be able to overide the client-specific from outside
>> > the client too.
>>
>> IMHO, the problem with evdev is, that it doesn't provide device
>> classes. The only class we have is "this is an input device". All
>> other event-flags can be combined in whatever way we want.
>>
>> So like 10 years ago when the first gamepad driver was introduced, we
>> added some mapping that was unique to this device (the device was
>> probably unique, too). Some time later, we added some other
>> gamepad-like driver with a different mapping (as it was probably a
>> very different device-type, back then, and we didn't see it coming
>> that this will become a wide-spread device-type).
>> However, today we notice that a "GamePad" is an established type of
>> device (like a touchpad), but we have tons of different mappings in
>> the kernel for backwards-compatibility reasons. I can see that this
>> kind of development can happen again (and very likely it _will_ happen
>> again) and it will happen for all kinds of devices.
>>
>> But that's why I designed the proposal from a compositor's view
>> instead of from a kernel's view.
>>
>> A touchpad driver of the compositor needs to know exactly what kind of
>> events it gets from the kernel. If it gets wrong events, it will
>> misbehave. As we cannot guarantee that all kernel drivers behave the
>> same way, the compositor's touchpad driver needs to work around all
>> these little details on a per-device basis.
>> To avoid this, I tried to abstract the touchpad-protocol and moved
>> per-device handling into a separate library. It detects all devices
>> that can serve as a touchpad and fixes trivial (1-to-1 mapping)
>> incompatibilities. This removes all per-device handling from the
>> touchpad driver and it can expect all input it gets to be conform with
>> a "touchpad" protocol.
>> And in fact, it removes this from all the compositor's input drivers.
>> So I think of it more like a "lib-detect-and-make-compat".
>>
>> All devices that do not fall into one of the categories (I called it
>> capability), will be handled as custom devices. So if we want an input
>> driver for a new fancy device, then we need a custom driver, anyway
>> (or adjust a generic driver to handle both). If at some point it turns
>> out, that this kind of device becomes more established, we can add a
>> new capability for it. Or we try extending an existing capability in a
>> backwards-compatible way. We can then remove the "custom-device
>> handling" from the input-driver and instead extend/write a generic
>> driver for the new capability.
>>
>>
>> So I cannot follow how you think this will have the same problems as
>> evdev? Or, let's ask the inverse question: How does this differ from
>> the X11 model where we move the "custom device handling" into the
>> drivers?
>
> first of all - I do agree with all of the above and I am certainly not
> against the idea of such a layer. (also don't discard the idea that I'm not
> looking at this the right way, I've been staring at X for too long and at
> wayland for not long enough)
>
> X divided input devices into pointers and keyboards. then we got complex
> devices, so we have XI, then XI2. XI supports simple device tagging
> tagging what a device actually is (e.g. as trackball vs a
> mouse). it didn't affect the data at all, it was just a property on the
> device ('type' in XDeviceInfo).

That's what I am trying with this library, but avoid the "generic" XI2
class. So instead of providing a class where all devices can be
represented in, I try an approach where we can add new classes
whenever we feel like it. But see below.

> where the whole lot started breaking down was having devices that were
> neither, or both, but having the need to classify them as either. this led
> to issues where keyboards didn't send XKB notifications because they ended
> up classified as pointer, etc.

Exactly, that's why I prefer tags over classes. We can add multiple
tags on devices, but normally put a device in only one class.

> wayland effectively divides devices into pointers, keyboards and touch
> devices and one way or the other you'll have to classify your device. since
> wayland doesnt (yet) expose the physical devices this is less of a problem
> than it is in X.
>
> so I guess my guestion is: how does this lib address this issue, especially
> for combined devices that have multiple capabilities that don't overlap
> well?
> e.g. a device hat has pointer buttons and gamepad buttons? I get it'll just
> have multiple capabilities, but how does this affect event routing?
>
> another thing, just to make it clearer for me: who is the consumer and the
> provider of this lib?
>
> What I envision for wayland is a structure like this:
>
>                 compositor
>                    |
>  libtouchpad libgeneric libjoystick libdigitizer ...
>                    |
>               libevdev_read
>                    |
>                  kernel
>
> (libevdev_read is a layer to read events from the device and handle
> disconnect/connect as well as SYN_DROPPED etc. so these don't need to be
> re-implemented everywhere)
>
> does libinputmapper come in between evdev_read and the modules?

Yes, exactly!

> last comment: the wayland protocol atm is fairly simple, it doesn't concern
> itself with raw data (yet). the xorg synaptics driver is complicated because
> mapping the various touchpads into a "touchpad protocol" is hard and I
> wouldn't even know where it would sit. We don't have a touchpad API between
> input module and server because it's just a stream of pointer events.
>
> So if you could explain where in the stack libinputmapper would sit that'd
> help me a lot because now I feel like I'm not getting something.

Yes, libinputmapper belongs in between driver-libraries and the
kernel. I am not sure where to place libevdev_read as handling
SYN_DROPPED and the like is very device-dependent. You can recover
from missing REL_* events more easily than from missing button-down
events. And you can ignore missing BTN_* events from gamepads, but you
should try to read them via EVIOCGKEY from keyboards to update
modifier-state.

This library is intended to solve the classification/detection
problem. While the kernel evdev-interface provides us a bunch of
information for each device, it doesn't provide any classification of
the device (mostly because it would be unreliable). So libinputmapper
as I see it is supposed to replace the heuristics that we currently
have (KEY_A-KEY_Z => keyboard, REL_X/Y => mouse, ...) with a more
user-controlled interface.

So a *compositor* defines the type of devices, that it wants to
support. As an example, it needs mouse+touchpad devices as
mouse-input, keyboards as keyboard input. It then listens for all
input hotplug events via udev, calls libinputmapper to provide tags
for the devices and then checks, whether it's one of the devices that
it wants. Client-side support is not involved, yet.

While device-classification might seem simple, there is a reason why
you guys wrote the highly customizable X11 InputDevice configuration
infrastructure. We cannot be sure that we do it right! And we should
never claim that. So libinputmapper comes with a configuration-parser
that allows users to tag devices manually. Example: If I have an
external Numpad with KEY_0 to KEY_9, but it's not correctly detected
as such, I could easily tag this device as KEYBOARD in a config-file
and my compositor would pick it up because libinputmapper now marks it
as KEYBOARD.

This removes the need of device-specific quirks in the
driver-libraries itself. But if we allow users to re-tag custom
devices, we must also guarantee that the driver understands the
device-input. That's why I came up with the mapping-tables. So we
_define_ how each device must behave to be tagged as such and the
driver-library can then simply expect every device to comply with
these rules (ABS_X/Y for TOUCHPAD devices means touch-position, for
example, not accelerometer data or anything else). But to avoid being
too restrictive, we allow simple 1-to-1 remappings for devices, that
report the same data but with different codes (ABS_RX/RY instead of
ABS_X/Y in the TOUCHPAD example).


So libinputmapper is used by a _compositor_. It's called to get device
tags and then the compositor forwards the data it reads to the correct
driver-library: keyboard data to libxkbcommon, touchpad data to
libtouchpadcommon, ... and so on. The optional mapping tables would be
applied by the compositor before forwarding to the driver-library, so
the driver-library gets "standardized" input.
And all these libraries expect a specific set of input events. It
might be hard to figure out and write down how each protocol works, so
expect this to be more like a "trial and error" definition. We don't
add tags that we have no use-case for (use case being a driver
library). And then we define devices to be compatible to a given tag,
if the device works with the driver as expected.


Special case #1:
A device has an integrated gamepad _and_ keyboard. We would then tag
it as GAMEPAD _and_ KEYBOARD. libxkbcommon would parse the keyboard
data, while "libgamepadcommon" would take care of gamepad-data.
Nothing special here, except that _if_ we add wl_gamepad, a client
would have no chance to see that both data comes from the same device.
But that's more a wayland-specific problem than a libinputmapper
problem. We could easily add a dummy wl_device object and link to it
from wl_gamepad and wl_keyboard so clients see the common ancestor.
(or a string-ID for each object, or sysfs path, or whatever we think
works best)

Special case #2:
A device that provides input that can be interpreted as absolute
mouse-position (exclusive-)or relative touchpad motion-events (eg.,
the Wii Remote IR-cam). We wouldn't tag this device at all, because it
has no definite interpretation.
Or, perhaps, we could tag it by default as one of both. However, we
would never tag it as both simultaneously. Because the device cannot
fulfill both definitions at the same time.
But a user could easily write some configuration to overwrite the
default tags without requiring any special device driver.

Special case #3:
A custom device that reports movement data that we can use to
calculate mouse movements (eg., Accelerometer/gyro data on some
remotes). We wouldn't tag this device at all, because we have no
driver to control it. So there is no other way than writing a custom
driver for it which transforms the custom input events into
mouse-movements so a compositor can use it. But this is outside the
scope of device-detection, and such libinputmapper.
However, if it turns out there is more than one device that reports
this data, we can add an INMAP_CAP_XY tag and use it to tag all such
devices so our new driver can use them.


Last but not least, if we end up with wl_custom_input as simple
evdev-event-stream for every device that the compositor is not
interested in, a client could then use libinputmapper on this object
to do what the compositor did for all other devices. So this is in no
way limited to the evdev interface as long as libinputmapper can
collect enough information to classify the device.


I hope that far too long text makes it more clear.
Cheers
David