[RFC] libinputmapper: Input device configuration for graphic-servers

[David Herrmann]

Hi

On Thu, May 23, 2013 at 7:32 AM, Peter Hutterer
<peter.hutterer at who-t.net> wrote:
> On Tue, May 21, 2013 at 04:30:03PM +0200, David Herrmann wrote:
>> Hi Peter
>>
>> On Tue, May 21, 2013 at 6:37 AM, Peter Hutterer
>> <peter.hutterer at who-t.net> wrote:
>> > On Thu, May 16, 2013 at 03:16:11PM +0200, David Herrmann wrote:
>> >> Hi Peter
>> >>
>> >> On Thu, May 16, 2013 at 7:37 AM, Peter Hutterer
>> >> <peter.hutterer at who-t.net> wrote:
>> >> > On Sun, May 12, 2013 at 04:20:59PM +0200, David Herrmann wrote:
>> >> [..]
>> >> >> So what is the proposed solution?
>> >> >> My recommendation is, that compositors still search for devices via
>> >> >> udev and use device drivers like libxkbcommon. So linux evdev handling
>> >> >> is still controlled by the compositor. However, I'd like to see
>> >> >> something like my libinputmapper proposal being used for device
>> >> >> detection and classification.
>> >> >>
>> >> >> libinputmapper provides an "inmap_evdev" object which reads device
>> >> >> information from an evdev-fd or sysfs /sys/class/input/input<num>
>> >> >> path, performs some heuristics to classify it and searches it's global
>> >> >> database for known fixups for broken devices.
>> >> >> It then provides "capabilities" to the caller, which allow them to see
>> >> >> what drivers to load on the device. And it provides a very simple
>> >> >> mapping table that allows to apply fixup mappings for broken devices.
>> >> >> These mappings are simple 1-to-1 mappings that are supposed to be
>> >> >> applied before drivers handle the input. This is to avoid
>> >> >> device-specific fixup in the drivers and move all this to the
>> >> >> inputmapper. An example would be a remapping for gamepads that report
>> >> >> BTN_A instead of BTN_NORTH, but we cannot fix them in the kernel for
>> >> >> backwards-compatibility reasons. The gamepad-driver can then assume
>> >> >> that if it receives BTN_NORTH, it is guaranteed to be BTN_NORTH and
>> >> >> doesn't need to special case xbox360/etc. controllers, because they're
>> >> >> broken.
>> >> >
>> >> > I think evdev is exactly that interface and apparently it doesn't work.
>> >> >
>> >> > if you want a mapping table, you need a per-client table because sooner or
>> >> > later you have a client that needs BTN_FOO when the kernel gives you BTN_BAR
>> >> > and you can't change the client to fix it.
>> >> >
>> >> > i.e. the same issue evdev has now, having a global remapping table just
>> >> > moves the problem down by 2 years.
>> >> >
>> >> > a mapping table is good, but you probably want two stages of mapping: one
>> >> > that's used in the compositor for truly broken devices that for some reason
>> >> > can't be fixed in the kernel, and one that's used on a per-client basis. and
>> >> > you'll likely want to be able to overide the client-specific from outside
>> >> > the client too.
>> >>
>> >> IMHO, the problem with evdev is, that it doesn't provide device
>> >> classes. The only class we have is "this is an input device". All
>> >> other event-flags can be combined in whatever way we want.
>> >>
>> >> So like 10 years ago when the first gamepad driver was introduced, we
>> >> added some mapping that was unique to this device (the device was
>> >> probably unique, too). Some time later, we added some other
>> >> gamepad-like driver with a different mapping (as it was probably a
>> >> very different device-type, back then, and we didn't see it coming
>> >> that this will become a wide-spread device-type).
>> >> However, today we notice that a "GamePad" is an established type of
>> >> device (like a touchpad), but we have tons of different mappings in
>> >> the kernel for backwards-compatibility reasons. I can see that this
>> >> kind of development can happen again (and very likely it _will_ happen
>> >> again) and it will happen for all kinds of devices.
>> >>
>> >> But that's why I designed the proposal from a compositor's view
>> >> instead of from a kernel's view.
>> >>
>> >> A touchpad driver of the compositor needs to know exactly what kind of
>> >> events it gets from the kernel. If it gets wrong events, it will
>> >> misbehave. As we cannot guarantee that all kernel drivers behave the
>> >> same way, the compositor's touchpad driver needs to work around all
>> >> these little details on a per-device basis.
>> >> To avoid this, I tried to abstract the touchpad-protocol and moved
>> >> per-device handling into a separate library. It detects all devices
>> >> that can serve as a touchpad and fixes trivial (1-to-1 mapping)
>> >> incompatibilities. This removes all per-device handling from the
>> >> touchpad driver and it can expect all input it gets to be conform with
>> >> a "touchpad" protocol.
>> >> And in fact, it removes this from all the compositor's input drivers.
>> >> So I think of it more like a "lib-detect-and-make-compat".
>> >>
>> >> All devices that do not fall into one of the categories (I called it
>> >> capability), will be handled as custom devices. So if we want an input
>> >> driver for a new fancy device, then we need a custom driver, anyway
>> >> (or adjust a generic driver to handle both). If at some point it turns
>> >> out, that this kind of device becomes more established, we can add a
>> >> new capability for it. Or we try extending an existing capability in a
>> >> backwards-compatible way. We can then remove the "custom-device
>> >> handling" from the input-driver and instead extend/write a generic
>> >> driver for the new capability.
>> >>
>> >>
>> >> So I cannot follow how you think this will have the same problems as
>> >> evdev? Or, let's ask the inverse question: How does this differ from
>> >> the X11 model where we move the "custom device handling" into the
>> >> drivers?
>> >
>> > first of all - I do agree with all of the above and I am certainly not
>> > against the idea of such a layer. (also don't discard the idea that I'm not
>> > looking at this the right way, I've been staring at X for too long and at
>> > wayland for not long enough)
>> >
>> > X divided input devices into pointers and keyboards. then we got complex
>> > devices, so we have XI, then XI2. XI supports simple device tagging
>> > tagging what a device actually is (e.g. as trackball vs a
>> > mouse). it didn't affect the data at all, it was just a property on the
>> > device ('type' in XDeviceInfo).
>>
>> That's what I am trying with this library, but avoid the "generic" XI2
>> class. So instead of providing a class where all devices can be
>> represented in, I try an approach where we can add new classes
>> whenever we feel like it. But see below.
>>
>> > where the whole lot started breaking down was having devices that were
>> > neither, or both, but having the need to classify them as either. this led
>> > to issues where keyboards didn't send XKB notifications because they ended
>> > up classified as pointer, etc.
>>
>> Exactly, that's why I prefer tags over classes. We can add multiple
>> tags on devices, but normally put a device in only one class.
>>
>> > wayland effectively divides devices into pointers, keyboards and touch
>> > devices and one way or the other you'll have to classify your device. since
>> > wayland doesnt (yet) expose the physical devices this is less of a problem
>> > than it is in X.
>> >
>> > so I guess my guestion is: how does this lib address this issue, especially
>> > for combined devices that have multiple capabilities that don't overlap
>> > well?
>> > e.g. a device hat has pointer buttons and gamepad buttons? I get it'll just
>> > have multiple capabilities, but how does this affect event routing?
>> >
>> > another thing, just to make it clearer for me: who is the consumer and the
>> > provider of this lib?
>> >
>> > What I envision for wayland is a structure like this:
>> >
>> >                 compositor
>> >                    |
>> >  libtouchpad libgeneric libjoystick libdigitizer ...
>> >                    |
>> >               libevdev_read
>> >                    |
>> >                  kernel
>> >
>> > (libevdev_read is a layer to read events from the device and handle
>> > disconnect/connect as well as SYN_DROPPED etc. so these don't need to be
>> > re-implemented everywhere)
>> >
>> > does libinputmapper come in between evdev_read and the modules?
>>
>> Yes, exactly!
>>
>> > last comment: the wayland protocol atm is fairly simple, it doesn't concern
>> > itself with raw data (yet). the xorg synaptics driver is complicated because
>> > mapping the various touchpads into a "touchpad protocol" is hard and I
>> > wouldn't even know where it would sit. We don't have a touchpad API between
>> > input module and server because it's just a stream of pointer events.
>> >
>> > So if you could explain where in the stack libinputmapper would sit that'd
>> > help me a lot because now I feel like I'm not getting something.
>>
>> Yes, libinputmapper belongs in between driver-libraries and the
>> kernel. I am not sure where to place libevdev_read as handling
>> SYN_DROPPED and the like is very device-dependent. You can recover
>> from missing REL_* events more easily than from missing button-down
>> events. And you can ignore missing BTN_* events from gamepads, but you
>> should try to read them via EVIOCGKEY from keyboards to update
>> modifier-state.
>
> I haven't even started with this yet, but my thought was that evdev_read
> transparently handles SYN_DROPPED by simply refreshing the state after a
> SYN_DROPPED and sending the missing events on to the module - as if they'd
> be coming from the driver. could be a bad idea, could be useful, I don't
> know yet :)

That doesn't work. For instance, keyboard shortcuts depend on the
order keys are pressed. If you read the state and send it to the
client, you might send a modifier-down event first, followed by a
key-down event, even though they could be generated in reversed order
(triggering a shortcut even though we shouldn't).

We can provide an evdev_recover() or evdev_refresh() helper, but the
application still needs to know when a SYN_DROPPED happened and handle
it correctly. But this is out-of-scope here..

>> This library is intended to solve the classification/detection
>> problem. While the kernel evdev-interface provides us a bunch of
>> information for each device, it doesn't provide any classification of
>> the device (mostly because it would be unreliable). So libinputmapper
>> as I see it is supposed to replace the heuristics that we currently
>> have (KEY_A-KEY_Z => keyboard, REL_X/Y => mouse, ...) with a more
>> user-controlled interface.
>>
>> So a *compositor* defines the type of devices, that it wants to
>> support. As an example, it needs mouse+touchpad devices as
>> mouse-input, keyboards as keyboard input. It then listens for all
>> input hotplug events via udev, calls libinputmapper to provide tags
>> for the devices and then checks, whether it's one of the devices that
>> it wants. Client-side support is not involved, yet.
>
> ok, that makes sense to me and is a good idea. it's a combined version of
> (formerly) udev's input_id tool and the Match* system that X has, in the
> form of a library with simple adjustment based on textfiles.
>
> good. and sorry, I did somewhat tie all this in with the client side as
> well, which is where the confusion came from.
>
> I think the tagging library will be quite private to the compositor
> implementation (even if that means all compositors). in the X case, we have
> evdev and the wacom driver for tablet devices since they're different
> enough to basic tablets. I suspect something like that
> may be required for wayland sooner or later.

input_id+Match* seems to be a good description. That's what I had in
mind. But I don't see a reason to keep it private to the compositor.
The compositor is the #1 user of the library, but whatever API is used
to forward events to a client, we shouldn't hide this source. We can
save all other kinds of useful information in these files. A client
could retrieve device-geometry information or other static data.
I designed the configuration to allow a global database with local
additions. So a compositor can have a different configuration than a
client. But both have the same global base/fallback configuration.

> so the tagging library would have to tag a device not just as graphics
> tablet but more as input module selection library. which again, means it's a
> library version of input_id and Match*, etc.
>
> what you will also need then is for drivers to provide hooks to notify what
> they can deal with. i.e. libtouchpadcommon needs to install data files for
> libinputmapper to notify that it's now taking over touchpads.

I wanted to keep this private to the compositor. So the compositor has
a list of drivers and a libinputmapper context. It then retrieves the
tags for a device and assigns the device depending on the tags to the
right driver. So libinputmapper always returns the same information.
It's up to the compositor to use this according to it's capabilities.
Or what would be the benefit of telling libinputmapper which drivers
are available?

I also don't understand what you mean by "install data files". What
data-files would libtouchpadcommon install?
My idea was to keep config-files generic. One file with "match"-rules
and a bunch of "device"-rules files. The match-rules would be used by
libinputmapper to find the device-rule (if any) files that should be
"applied" to the device. The device-rules contain different sections.
libinputmapper only parses one section which affects tagging. This is
optional and only needed if the default tagging-heuristics are wrong
for that device.
All other information in the device-rules files are ignored, but we
can allow the compositor to retrieve the file-name. It can then pass
the file-name to libtouchpadcommon which can then look whether the
user specified a [libtouchpadcommon] section for the device. If not,
it uses the default values.

configuration example:
Four files, a "match"-file and 3 device-rules. It assigns
"wiimote.rule" to all Wii-Remote input devices so they are ignored
(which is what we do for Xorg, too). And it provides a custom rule for
a specific device which can be installed by the user. It doesn't
overwrite tagging-rules, so the defaults are applied. But it provides
additional information for xkbcommon to set the layout different for
this device.
The third file is matched on every device and causes an "en" layout
for xkbcommon to be applied.
(The match-logic can be extended, this is just an example that assumes
we stop on the first match..)

file: inputmapper.match:
[match]
name = Nintendo Wii Remote*
rule = gamepad/wiimote.rule
[match]
vid = 0x3663
pid = 0x0010
rule = misc/custom.rule
[match]
rule = default.rule

file: gamepad/wiimote.rule
[tags]
# clear tags to avoid using Wii-Remote IR/Accel-data erroneously as mouse-input
set =

file: misc/custom.rule
[xkbcommon]
layout = de
variant = nodeadkeys

file: default.rule
[xkbcommon]
layout = en

>> While device-classification might seem simple, there is a reason why
>> you guys wrote the highly customizable X11 InputDevice configuration
>> infrastructure. We cannot be sure that we do it right! And we should
>> never claim that. So libinputmapper comes with a configuration-parser
>> that allows users to tag devices manually. Example: If I have an
>> external Numpad with KEY_0 to KEY_9, but it's not correctly detected
>> as such, I could easily tag this device as KEYBOARD in a config-file
>> and my compositor would pick it up because libinputmapper now marks it
>> as KEYBOARD.
>>
>> This removes the need of device-specific quirks in the
>> driver-libraries itself. But if we allow users to re-tag custom
>> devices, we must also guarantee that the driver understands the
>> device-input. That's why I came up with the mapping-tables. So we
>> _define_ how each device must behave to be tagged as such and the
>> driver-library can then simply expect every device to comply with
>> these rules (ABS_X/Y for TOUCHPAD devices means touch-position, for
>> example, not accelerometer data or anything else). But to avoid being
>> too restrictive, we allow simple 1-to-1 remappings for devices, that
>> report the same data but with different codes (ABS_RX/RY instead of
>> ABS_X/Y in the TOUCHPAD example).
>
> while there is quite a bit of overlap, I'm not convinced yet that a mapping
> library and a tagging library are the same thing - or should be combined.
> the two issues are imo orthogonal and though both would use a device
> database, one is tied more to the compositor's needs (i.e. which driver) and
> one to truly fix up kernel issues (could thus be useful for X too).
> this doesn't change the flow of what you describe in the next paragraph
> below, merely whether it's one or two components involved.

Yes, you are right. The mapping makes more sense as part of libevdev_read.
And maybe libinputmapper should rather be named something like
"device-database", libdevdb? Because what it does is basically provide
additional information per device that the kernel cannot provide.

Cheers
David