[RFC] libinputmapper: Input device configuration for graphic-servers

[Peter Hutterer]

On Tue, May 21, 2013 at 04:30:03PM +0200, David Herrmann wrote:
> Hi Peter
> 
> On Tue, May 21, 2013 at 6:37 AM, Peter Hutterer
> <peter.hutterer at who-t.net> wrote:
> > On Thu, May 16, 2013 at 03:16:11PM +0200, David Herrmann wrote:
> >> Hi Peter
> >>
> >> On Thu, May 16, 2013 at 7:37 AM, Peter Hutterer
> >> <peter.hutterer at who-t.net> wrote:
> >> > On Sun, May 12, 2013 at 04:20:59PM +0200, David Herrmann wrote:
> >> [..]
> >> >> So what is the proposed solution?
> >> >> My recommendation is, that compositors still search for devices via
> >> >> udev and use device drivers like libxkbcommon. So linux evdev handling
> >> >> is still controlled by the compositor. However, I'd like to see
> >> >> something like my libinputmapper proposal being used for device
> >> >> detection and classification.
> >> >>
> >> >> libinputmapper provides an "inmap_evdev" object which reads device
> >> >> information from an evdev-fd or sysfs /sys/class/input/input<num>
> >> >> path, performs some heuristics to classify it and searches it's global
> >> >> database for known fixups for broken devices.
> >> >> It then provides "capabilities" to the caller, which allow them to see
> >> >> what drivers to load on the device. And it provides a very simple
> >> >> mapping table that allows to apply fixup mappings for broken devices.
> >> >> These mappings are simple 1-to-1 mappings that are supposed to be
> >> >> applied before drivers handle the input. This is to avoid
> >> >> device-specific fixup in the drivers and move all this to the
> >> >> inputmapper. An example would be a remapping for gamepads that report
> >> >> BTN_A instead of BTN_NORTH, but we cannot fix them in the kernel for
> >> >> backwards-compatibility reasons. The gamepad-driver can then assume
> >> >> that if it receives BTN_NORTH, it is guaranteed to be BTN_NORTH and
> >> >> doesn't need to special case xbox360/etc. controllers, because they're
> >> >> broken.
> >> >
> >> > I think evdev is exactly that interface and apparently it doesn't work.
> >> >
> >> > if you want a mapping table, you need a per-client table because sooner or
> >> > later you have a client that needs BTN_FOO when the kernel gives you BTN_BAR
> >> > and you can't change the client to fix it.
> >> >
> >> > i.e. the same issue evdev has now, having a global remapping table just
> >> > moves the problem down by 2 years.
> >> >
> >> > a mapping table is good, but you probably want two stages of mapping: one
> >> > that's used in the compositor for truly broken devices that for some reason
> >> > can't be fixed in the kernel, and one that's used on a per-client basis. and
> >> > you'll likely want to be able to overide the client-specific from outside
> >> > the client too.
> >>
> >> IMHO, the problem with evdev is, that it doesn't provide device
> >> classes. The only class we have is "this is an input device". All
> >> other event-flags can be combined in whatever way we want.
> >>
> >> So like 10 years ago when the first gamepad driver was introduced, we
> >> added some mapping that was unique to this device (the device was
> >> probably unique, too). Some time later, we added some other
> >> gamepad-like driver with a different mapping (as it was probably a
> >> very different device-type, back then, and we didn't see it coming
> >> that this will become a wide-spread device-type).
> >> However, today we notice that a "GamePad" is an established type of
> >> device (like a touchpad), but we have tons of different mappings in
> >> the kernel for backwards-compatibility reasons. I can see that this
> >> kind of development can happen again (and very likely it _will_ happen
> >> again) and it will happen for all kinds of devices.
> >>
> >> But that's why I designed the proposal from a compositor's view
> >> instead of from a kernel's view.
> >>
> >> A touchpad driver of the compositor needs to know exactly what kind of
> >> events it gets from the kernel. If it gets wrong events, it will
> >> misbehave. As we cannot guarantee that all kernel drivers behave the
> >> same way, the compositor's touchpad driver needs to work around all
> >> these little details on a per-device basis.
> >> To avoid this, I tried to abstract the touchpad-protocol and moved
> >> per-device handling into a separate library. It detects all devices
> >> that can serve as a touchpad and fixes trivial (1-to-1 mapping)
> >> incompatibilities. This removes all per-device handling from the
> >> touchpad driver and it can expect all input it gets to be conform with
> >> a "touchpad" protocol.
> >> And in fact, it removes this from all the compositor's input drivers.
> >> So I think of it more like a "lib-detect-and-make-compat".
> >>
> >> All devices that do not fall into one of the categories (I called it
> >> capability), will be handled as custom devices. So if we want an input
> >> driver for a new fancy device, then we need a custom driver, anyway
> >> (or adjust a generic driver to handle both). If at some point it turns
> >> out, that this kind of device becomes more established, we can add a
> >> new capability for it. Or we try extending an existing capability in a
> >> backwards-compatible way. We can then remove the "custom-device
> >> handling" from the input-driver and instead extend/write a generic
> >> driver for the new capability.
> >>
> >>
> >> So I cannot follow how you think this will have the same problems as
> >> evdev? Or, let's ask the inverse question: How does this differ from
> >> the X11 model where we move the "custom device handling" into the
> >> drivers?
> >
> > first of all - I do agree with all of the above and I am certainly not
> > against the idea of such a layer. (also don't discard the idea that I'm not
> > looking at this the right way, I've been staring at X for too long and at
> > wayland for not long enough)
> >
> > X divided input devices into pointers and keyboards. then we got complex
> > devices, so we have XI, then XI2. XI supports simple device tagging
> > tagging what a device actually is (e.g. as trackball vs a
> > mouse). it didn't affect the data at all, it was just a property on the
> > device ('type' in XDeviceInfo).
> 
> That's what I am trying with this library, but avoid the "generic" XI2
> class. So instead of providing a class where all devices can be
> represented in, I try an approach where we can add new classes
> whenever we feel like it. But see below.
> 
> > where the whole lot started breaking down was having devices that were
> > neither, or both, but having the need to classify them as either. this led
> > to issues where keyboards didn't send XKB notifications because they ended
> > up classified as pointer, etc.
> 
> Exactly, that's why I prefer tags over classes. We can add multiple
> tags on devices, but normally put a device in only one class.
> 
> > wayland effectively divides devices into pointers, keyboards and touch
> > devices and one way or the other you'll have to classify your device. since
> > wayland doesnt (yet) expose the physical devices this is less of a problem
> > than it is in X.
> >
> > so I guess my guestion is: how does this lib address this issue, especially
> > for combined devices that have multiple capabilities that don't overlap
> > well?
> > e.g. a device hat has pointer buttons and gamepad buttons? I get it'll just
> > have multiple capabilities, but how does this affect event routing?
> >
> > another thing, just to make it clearer for me: who is the consumer and the
> > provider of this lib?
> >
> > What I envision for wayland is a structure like this:
> >
> >                 compositor
> >                    |
> >  libtouchpad libgeneric libjoystick libdigitizer ...
> >                    |
> >               libevdev_read
> >                    |
> >                  kernel
> >
> > (libevdev_read is a layer to read events from the device and handle
> > disconnect/connect as well as SYN_DROPPED etc. so these don't need to be
> > re-implemented everywhere)
> >
> > does libinputmapper come in between evdev_read and the modules?
> 
> Yes, exactly!
> 
> > last comment: the wayland protocol atm is fairly simple, it doesn't concern
> > itself with raw data (yet). the xorg synaptics driver is complicated because
> > mapping the various touchpads into a "touchpad protocol" is hard and I
> > wouldn't even know where it would sit. We don't have a touchpad API between
> > input module and server because it's just a stream of pointer events.
> >
> > So if you could explain where in the stack libinputmapper would sit that'd
> > help me a lot because now I feel like I'm not getting something.
> 
> Yes, libinputmapper belongs in between driver-libraries and the
> kernel. I am not sure where to place libevdev_read as handling
> SYN_DROPPED and the like is very device-dependent. You can recover
> from missing REL_* events more easily than from missing button-down
> events. And you can ignore missing BTN_* events from gamepads, but you
> should try to read them via EVIOCGKEY from keyboards to update
> modifier-state.

I haven't even started with this yet, but my thought was that evdev_read
transparently handles SYN_DROPPED by simply refreshing the state after a
SYN_DROPPED and sending the missing events on to the module - as if they'd
be coming from the driver. could be a bad idea, could be useful, I don't
know yet :)

> This library is intended to solve the classification/detection
> problem. While the kernel evdev-interface provides us a bunch of
> information for each device, it doesn't provide any classification of
> the device (mostly because it would be unreliable). So libinputmapper
> as I see it is supposed to replace the heuristics that we currently
> have (KEY_A-KEY_Z => keyboard, REL_X/Y => mouse, ...) with a more
> user-controlled interface.
> 
> So a *compositor* defines the type of devices, that it wants to
> support. As an example, it needs mouse+touchpad devices as
> mouse-input, keyboards as keyboard input. It then listens for all
> input hotplug events via udev, calls libinputmapper to provide tags
> for the devices and then checks, whether it's one of the devices that
> it wants. Client-side support is not involved, yet.

ok, that makes sense to me and is a good idea. it's a combined version of
(formerly) udev's input_id tool and the Match* system that X has, in the
form of a library with simple adjustment based on textfiles.

good. and sorry, I did somewhat tie all this in with the client side as
well, which is where the confusion came from.

I think the tagging library will be quite private to the compositor
implementation (even if that means all compositors). in the X case, we have
evdev and the wacom driver for tablet devices since they're different
enough to basic tablets. I suspect something like that
may be required for wayland sooner or later.

so the tagging library would have to tag a device not just as graphics
tablet but more as input module selection library. which again, means it's a
library version of input_id and Match*, etc.

what you will also need then is for drivers to provide hooks to notify what
they can deal with. i.e. libtouchpadcommon needs to install data files for
libinputmapper to notify that it's now taking over touchpads.

> While device-classification might seem simple, there is a reason why
> you guys wrote the highly customizable X11 InputDevice configuration
> infrastructure. We cannot be sure that we do it right! And we should
> never claim that. So libinputmapper comes with a configuration-parser
> that allows users to tag devices manually. Example: If I have an
> external Numpad with KEY_0 to KEY_9, but it's not correctly detected
> as such, I could easily tag this device as KEYBOARD in a config-file
> and my compositor would pick it up because libinputmapper now marks it
> as KEYBOARD.
> 
> This removes the need of device-specific quirks in the
> driver-libraries itself. But if we allow users to re-tag custom
> devices, we must also guarantee that the driver understands the
> device-input. That's why I came up with the mapping-tables. So we
> _define_ how each device must behave to be tagged as such and the
> driver-library can then simply expect every device to comply with
> these rules (ABS_X/Y for TOUCHPAD devices means touch-position, for
> example, not accelerometer data or anything else). But to avoid being
> too restrictive, we allow simple 1-to-1 remappings for devices, that
> report the same data but with different codes (ABS_RX/RY instead of
> ABS_X/Y in the TOUCHPAD example).

while there is quite a bit of overlap, I'm not convinced yet that a mapping
library and a tagging library are the same thing - or should be combined.
the two issues are imo orthogonal and though both would use a device
database, one is tied more to the compositor's needs (i.e. which driver) and
one to truly fix up kernel issues (could thus be useful for X too).
this doesn't change the flow of what you describe in the next paragraph
below, merely whether it's one or two components involved.

Cheers,
   Peter

> So libinputmapper is used by a _compositor_. It's called to get device
> tags and then the compositor forwards the data it reads to the correct
> driver-library: keyboard data to libxkbcommon, touchpad data to
> libtouchpadcommon, ... and so on. The optional mapping tables would be
> applied by the compositor before forwarding to the driver-library, so
> the driver-library gets "standardized" input.
> And all these libraries expect a specific set of input events. It
> might be hard to figure out and write down how each protocol works, so
> expect this to be more like a "trial and error" definition. We don't
> add tags that we have no use-case for (use case being a driver
> library). And then we define devices to be compatible to a given tag,
> if the device works with the driver as expected.
> 
> Special case #1:
> A device has an integrated gamepad _and_ keyboard. We would then tag
> it as GAMEPAD _and_ KEYBOARD. libxkbcommon would parse the keyboard
> data, while "libgamepadcommon" would take care of gamepad-data.
> Nothing special here, except that _if_ we add wl_gamepad, a client
> would have no chance to see that both data comes from the same device.
> But that's more a wayland-specific problem than a libinputmapper
> problem. We could easily add a dummy wl_device object and link to it
> from wl_gamepad and wl_keyboard so clients see the common ancestor.
> (or a string-ID for each object, or sysfs path, or whatever we think
> works best)
> 
> Special case #2:
> A device that provides input that can be interpreted as absolute
> mouse-position (exclusive-)or relative touchpad motion-events (eg.,
> the Wii Remote IR-cam). We wouldn't tag this device at all, because it
> has no definite interpretation.
> Or, perhaps, we could tag it by default as one of both. However, we
> would never tag it as both simultaneously. Because the device cannot
> fulfill both definitions at the same time.
> But a user could easily write some configuration to overwrite the
> default tags without requiring any special device driver.
> 
> Special case #3:
> A custom device that reports movement data that we can use to
> calculate mouse movements (eg., Accelerometer/gyro data on some
> remotes). We wouldn't tag this device at all, because we have no
> driver to control it. So there is no other way than writing a custom
> driver for it which transforms the custom input events into
> mouse-movements so a compositor can use it. But this is outside the
> scope of device-detection, and such libinputmapper.
> However, if it turns out there is more than one device that reports
> this data, we can add an INMAP_CAP_XY tag and use it to tag all such
> devices so our new driver can use them.
> 
> 
> Last but not least, if we end up with wl_custom_input as simple
> evdev-event-stream for every device that the compositor is not
> interested in, a client could then use libinputmapper on this object
> to do what the compositor did for all other devices. So this is in no
> way limited to the evdev interface as long as libinputmapper can
> collect enough information to classify the device.
> 
> 
> I hope that far too long text makes it more clear.
> Cheers
> David