[RFC] libinputmapper: Input device configuration for graphic-servers

[Peter Hutterer]

On Thu, May 16, 2013 at 03:16:11PM +0200, David Herrmann wrote:
> Hi Peter
> 
> On Thu, May 16, 2013 at 7:37 AM, Peter Hutterer
> <peter.hutterer at who-t.net> wrote:
> > On Sun, May 12, 2013 at 04:20:59PM +0200, David Herrmann wrote:
> [..]
> >> So what is the proposed solution?
> >> My recommendation is, that compositors still search for devices via
> >> udev and use device drivers like libxkbcommon. So linux evdev handling
> >> is still controlled by the compositor. However, I'd like to see
> >> something like my libinputmapper proposal being used for device
> >> detection and classification.
> >>
> >> libinputmapper provides an "inmap_evdev" object which reads device
> >> information from an evdev-fd or sysfs /sys/class/input/input<num>
> >> path, performs some heuristics to classify it and searches it's global
> >> database for known fixups for broken devices.
> >> It then provides "capabilities" to the caller, which allow them to see
> >> what drivers to load on the device. And it provides a very simple
> >> mapping table that allows to apply fixup mappings for broken devices.
> >> These mappings are simple 1-to-1 mappings that are supposed to be
> >> applied before drivers handle the input. This is to avoid
> >> device-specific fixup in the drivers and move all this to the
> >> inputmapper. An example would be a remapping for gamepads that report
> >> BTN_A instead of BTN_NORTH, but we cannot fix them in the kernel for
> >> backwards-compatibility reasons. The gamepad-driver can then assume
> >> that if it receives BTN_NORTH, it is guaranteed to be BTN_NORTH and
> >> doesn't need to special case xbox360/etc. controllers, because they're
> >> broken.
> >
> > I think evdev is exactly that interface and apparently it doesn't work.
> >
> > if you want a mapping table, you need a per-client table because sooner or
> > later you have a client that needs BTN_FOO when the kernel gives you BTN_BAR
> > and you can't change the client to fix it.
> >
> > i.e. the same issue evdev has now, having a global remapping table just
> > moves the problem down by 2 years.
> >
> > a mapping table is good, but you probably want two stages of mapping: one
> > that's used in the compositor for truly broken devices that for some reason
> > can't be fixed in the kernel, and one that's used on a per-client basis. and
> > you'll likely want to be able to overide the client-specific from outside
> > the client too.
> 
> IMHO, the problem with evdev is, that it doesn't provide device
> classes. The only class we have is "this is an input device". All
> other event-flags can be combined in whatever way we want.
> 
> So like 10 years ago when the first gamepad driver was introduced, we
> added some mapping that was unique to this device (the device was
> probably unique, too). Some time later, we added some other
> gamepad-like driver with a different mapping (as it was probably a
> very different device-type, back then, and we didn't see it coming
> that this will become a wide-spread device-type).
> However, today we notice that a "GamePad" is an established type of
> device (like a touchpad), but we have tons of different mappings in
> the kernel for backwards-compatibility reasons. I can see that this
> kind of development can happen again (and very likely it _will_ happen
> again) and it will happen for all kinds of devices.
> 
> But that's why I designed the proposal from a compositor's view
> instead of from a kernel's view.
> 
> A touchpad driver of the compositor needs to know exactly what kind of
> events it gets from the kernel. If it gets wrong events, it will
> misbehave. As we cannot guarantee that all kernel drivers behave the
> same way, the compositor's touchpad driver needs to work around all
> these little details on a per-device basis.
> To avoid this, I tried to abstract the touchpad-protocol and moved
> per-device handling into a separate library. It detects all devices
> that can serve as a touchpad and fixes trivial (1-to-1 mapping)
> incompatibilities. This removes all per-device handling from the
> touchpad driver and it can expect all input it gets to be conform with
> a "touchpad" protocol.
> And in fact, it removes this from all the compositor's input drivers.
> So I think of it more like a "lib-detect-and-make-compat".
> 
> All devices that do not fall into one of the categories (I called it
> capability), will be handled as custom devices. So if we want an input
> driver for a new fancy device, then we need a custom driver, anyway
> (or adjust a generic driver to handle both). If at some point it turns
> out, that this kind of device becomes more established, we can add a
> new capability for it. Or we try extending an existing capability in a
> backwards-compatible way. We can then remove the "custom-device
> handling" from the input-driver and instead extend/write a generic
> driver for the new capability.
> 
> 
> So I cannot follow how you think this will have the same problems as
> evdev? Or, let's ask the inverse question: How does this differ from
> the X11 model where we move the "custom device handling" into the
> drivers?

first of all - I do agree with all of the above and I am certainly not
against the idea of such a layer. (also don't discard the idea that I'm not
looking at this the right way, I've been staring at X for too long and at
wayland for not long enough)

X divided input devices into pointers and keyboards. then we got complex
devices, so we have XI, then XI2. XI supports simple device tagging 
tagging what a device actually is (e.g. as trackball vs a
mouse). it didn't affect the data at all, it was just a property on the
device ('type' in XDeviceInfo).

where the whole lot started breaking down was having devices that were
neither, or both, but having the need to classify them as either. this led
to issues where keyboards didn't send XKB notifications because they ended
up classified as pointer, etc.

wayland effectively divides devices into pointers, keyboards and touch
devices and one way or the other you'll have to classify your device. since
wayland doesnt (yet) expose the physical devices this is less of a problem
than it is in X.

so I guess my guestion is: how does this lib address this issue, especially
for combined devices that have multiple capabilities that don't overlap
well?
e.g. a device hat has pointer buttons and gamepad buttons? I get it'll just
have multiple capabilities, but how does this affect event routing?

another thing, just to make it clearer for me: who is the consumer and the
provider of this lib?

What I envision for wayland is a structure like this:

                compositor
                   |
 libtouchpad libgeneric libjoystick libdigitizer ...
                   |
              libevdev_read 
                   |
                 kernel 

(libevdev_read is a layer to read events from the device and handle
disconnect/connect as well as SYN_DROPPED etc. so these don't need to be
re-implemented everywhere)

does libinputmapper come in between evdev_read and the modules?

last comment: the wayland protocol atm is fairly simple, it doesn't concern
itself with raw data (yet). the xorg synaptics driver is complicated because
mapping the various touchpads into a "touchpad protocol" is hard and I
wouldn't even know where it would sit. We don't have a touchpad API between
input module and server because it's just a stream of pointer events. 

So if you could explain where in the stack libinputmapper would sit that'd
help me a lot because now I feel like I'm not getting something.

Cheers,
   Peter

> >> libinputmapper would use some static heuristics for all this, but
> >> additionally parse user-configuration. A configuration file contains
> >> "[match]" entries, which specify device-configurations to load on
> >> mtached devices. The device-configurations then contain different
> >> blocks which can overwrite the auto-detected capabilities or provide
> >> fixup-mappings.
> >> But other configuration-sections can also be provided in the same
> >> file. For instance an [xkb] group could be specified to set XKB
> >> layout/variant/options. This won't be parsed by libinputmapper, but it
> >> would provide the matching and forward the device-configuration to the
> >> caller, which can pass it on to the driver.
> >> Additionally, API users can provide separate (local)
> >> configuration-files that allow users to change the configuration for
> >> kmscon one way and another way for gnome-shell (or similar).
> >>
> >> One can think of these configurations to replace the old
> >> /etc/X11/xorg.conf.d/*.conf "InputClass" configuration items. Except
> >> that the "matching" is offloaded to a separate file, and the "to be
> >> applied configurations" are specified in "rule" files, that are linked
> >> to from the "match" configurations. See below for syntax-examples.
> >>
> >>
> >> I haven't implemented the library, yet. I am working on this. However,
> >> I wrote up an API proposal that I attached inline below. kmscon input
> >> handling is what I need it for, so I want something like this, anyway.
> >> But I thought other compositors might be interested in this, too.
> >>
> >> Comments are welcome!
> >> Regards
> >> David
> >>
> >>
> >
> > [...]
> >
> >> /**
> >>  * @enum inmap_capability
> >>  * Generic device capabilities
> >>  *
> >>  * Device capabilities describe what generic interface is provided by the
> >>  * detected device. A device can support any combination of them (or none).
> >>  * Each interface follows specific rules. If a device does not follow these
> >>  * rules, we must provide a mapping table so the correct data is reported.
> >>  *
> >>  * Capabilities can be added for all kinds of generic devices. However, custom
> >>  * devices with model-specific features do not belong here, as their drivers
> >>  * need to perform device-detection themselves, anyway.
> >>  *
> >>  * Furthermore, each capability needs a well-defined protocol. They are mostly
> >>  * already provided by the kernel. But there are many devices and drivers
> >>  * which provide inconsistent behavior. That's why each capability has to
> >>  * be consistent and coherent and misbehaving devices get fixup mappings, if
> >>  * we cannot fix it in the kernel device driver itself.
> >>  */
> >> enum inmap_capability {
> >> /**
> >> * @INMAP_CAP_KEYBOARD
> >> * Keyboard interface
> >> *
> >> * The device is a real keyboard that is used for normal input.
> >> * Power-buttons and similar devices which report KEY_* events for
> >> * non-keyboard buttons are not considered a keyboard in this sense.
> >> * Only real keyboards and small extension devices (like external
> >> * NumPads) belong here.
> >> * The defined KEY_* events are not listed here as normally no
> >> * mapping is applied. Libraries like libxkbcommon should be used
> >> * for keyboard handling and mapping.
> >> */
> >> INMAP_CAP_KEYBOARD = (1 << 0),
> >>
> >> /**
> >> * @INMAP_CAP_MOUSE
> >> * Mouse interface
> >> *
> >> * The device is a real mouse with relative motion events. Motion
> >> * is reported as REL_X/Y. Buttons are reported as BTN_0-BTN_9 or
> >> * BTN_LEFT-BTN_TASK.
> >> * Other mouse-like devices (especially those with absolute motion
> >> * events) do not belong here.
> >> */
> >> INMAP_CAP_MOUSE = (1 << 1),
> >>
> >> /**
> >> * @INMAP_CAP_TOUCHPAD
> >> * Touchpad interface
> >> *
> >> * Touchpad devices belong here. But only those that are used as
> >> * pointing devices and are intended to drive a mouse device. That is,
> >> * the input is interpreted as relative motion, even though the
> >> * reported data is normally absolute.
> >> * Hence, touchscreens do not belong here. They are normally
> >> * interpreted as absolute position input.
> >> *
> >> * For multitouch devices, libmtdev defines the protocol.
> >
> > kernel MT Procol B is a better term here, libmtdev merely converts A to B.
> 
> I actually meant that this includes all MT protocols (all A models and
> B), that is, eveything that mtdev accepts. A driver which uses this
> would then probably call mtdev to get B only. However, doing this
> conversion here would be useless, I think, because we already have a
> consistent interface with the 3 (or 4?) different MT protocols.
> 
> But I am also fine with calling mtdev directly here and defining this as MT-B.
> 
> >> *
> >> * @TODO define the protocol (see the synaptics descriptions)
> >> */
> >> INMAP_CAP_TOUCHPAD = (1 << 2),
> >>
> >> /**
> >> * @INMAP_CAP_TOUCHSCREEN
> >> * Touchscreen interface
> >> *
> >> * Touchscreens are exactly the same as @INMAP_CAP_TOUCHPAD, but they
> >> * are normally attached to a single monitor and interpreted as
> >> * absolute positioning input.
> >> * Therefore, they are not included in @INMAP_CAP_TOUCHPAD, but are
> >> * a separate group instead. The reported protocol is the same, though.
> >> *
> >> * @sa INMAP_CAP_TOUCHPAD
> >> */
> >> INMAP_CAP_TOUCHSCREEN = (1 << 3),
> >>
> >> /**
> >> * @INMAP_CAP_ACCELEROMETER
> >> * Accelerometer interface
> >> *
> >> * Accelerometer devices report linear acceleration data as ABS_X/Y/Z
> >> * and rotational acceleration as ABS_RX/Y/Z.
> >> *
> >> * @TODO this collides with ABS_X/Y of absolute pointing devices
> >> *       introduce ABS_ACCELX/Y/Z
> >> */
> >> INMAP_CAP_ACCELEROMETER = (1 << 4),
> >>
> >> /**
> >> * @INMAP_CAP_GAMEPAD
> >> * Gamepad interface
> >> *
> >> * All standard gamepad devices belong here.
> >> *
> >> * @TODO define gamepad mapping (see linux-input at vger discussion)
> >> */
> >> INMAP_CAP_GAMEPAD = (1 << 5),
> >>
> >> /**
> >> * @INMAP_CAP_JOYSTICK
> >> * Joystick interface
> >> *
> >> * All standard joystick devices belong here.
> >> *
> >> * @TODO define joystick mapping
> >> */
> >> INMAP_CAP_JOYSTICK = (1 << 6),
> >> };
> >
> > why are gamepads and joysticks different? buttons, a few axes that may or
> > may not map to x/y and the rest is device-specific.
> > this may be in the thread, but I still haven't gone through all msgs here.
> 
> Many ABS_* values are shared between joysticks and gamepads. So I
> either map all gamepads to values that don't collide with joysticks or
> I tell the application whether it's a joystick or gamepad. I think the
> latter is easier.
> 
> >
> > one missing section are graphics tablets but even there you could argue
> > they're just buttons, x/y and a few extra axes that need client-specific
> > handling.
> 
> I had DIGITIZER in here earlier, but removed it to keep it simpler. I
> expect new types to be added over time, so I thought we could add
> tablets later, too.
> 
> >> /**
> >>  * @defgroup context Library Context
> >>  * Managing library contexts
> >>  *
> >>  * A context object contains all top-level information. It can be shared
> >>  * between multiple device objects and provides global infrastructure to them.
> >>  *
> >>  * @{
> >>  */
> >>
> >> /**
> >>  * @enum inmap_context_flags
> >>  * Context creation and operation flags
> >>  *
> >>  * Different flags that control how a context is created and how it behaves.
> >>  *
> >>  * @memberof inmap_context
> >>  * @sa inmap_context_new()
> >>  */
> >> enum inmap_context_flags {
> >> /**
> >> * @INMAP_CONTEXT_IGNORE_ENVIRONMENT
> >> * Ignore environment variables
> >> *
> >> * The context object will not use any environment variables as
> >> * default values if they are not explicitly set.
> >> *
> >> * @memberof inmap_context
> >> */
> >> INMAP_CONTEXT_IGNORE_ENVIRONMENT = (1 << 0),
> >>
> >> /**
> >> * @INMAP_CONTEXT_MONITOR_CONF_FILES
> >> * Monitor configuration files
> >> *
> >> * If set, the context will monitor configuration files for runtime
> >> * modifications. This allows adapting to changes without restarting
> >> * the application.
> >> * Changes are not automatically applied. The application has to
> >> * react to these events and retrigger device detection for all
> >> * active devices.
> >> *
> >> * @memberof inmap_context
> >> * @sa inmap_context_dispatch()
> >> */
> >> INMAP_CONTEXT_MONITOR_CONF_FILES = (1 << 1),
> >
> > tbh, I'd make that a required behaviour.
> 
> That would mean libraries that just want to enumerate/list devices
> need this monitor. But I could at least make it the default and add a
> "NO_MONITOR" flag.
> 
> Thanks!
> David