[PATCH libinput 0/24] Tablet support

[Carlos Garnacho]

Hey Jason,

On jue, 2014-05-22 at 11:17 -0700, Jason Gerecke wrote:
> On Wed, May 21, 2014 at 10:17 PM, Chandler Paul <thatslyude at gmail.com> wrote:
> > Hi! Sorry this took so long to write, I've been spending a lot of my
> > time recently trying to understand the libinput code and all of that
> > good stuff, and I wanted to make sure I had a decent understanding of it
> > before I actually wrote up a response.
> >
> I'll take a look over the patches and see if I can provide any
> comments. I haven't looked at the libinput code before, so I'll
> probably be stumbling around for a bit :D
> 
> > On Mon, 2014-04-21 at 19:11 +0200, Carlos Garnacho wrote:
> >> Hey there!,
> >>
> >> Here's a few patches to have libinput handle events from tablets,
> >> these devices are basically pointer devices, with a varying range
> >> of extra buttons (either stylus or "pad" buttons), and extra ABS_*
> >> axes. These devices also often offer information about the stylus
> >> in use, and its BTN_TOOL_* codes.
> >>
> >> So I've gone for reusing and extending libinput_event_pointer, adding
> >> extra libinput_pointer_axis values, and adding an "axis_frame" event
> >> to mark the end of a set of simultaneous axis changes, and a "tool_update"
> >> event to mark tool changes (and delimit proximity). These features are
> >> only triggered if a new LIBINPUT_DEVICE_CAP_STYLUS capability is set.
> > I'm with Peter on splitting these up. It seems kind of inconsistent with
> > the rest of libinput (with what I've gathered, anyway). A
> > tablet-specific event sounds interesting too, but I feel like
> > compressing all of the axis changes into. For now I'm going to work on
> > having all the axis changes reported as separate POINTER_AXIS events.
> >
> >>
> >> Caveats:
> >>
> >> * Many of these devices have also tactile strips or wheels, but these are
> >>   unhandled so far... On the devices I've got available for testing, current
> >>   kernel support seems varyingly inconsistent:
> >>
> >>   - One device has 2 strips, which report on ABS_RX/RY (radial??). Min/max
> >>     are 0..4096, but the reported values are 1,2,4,8,16... So effectively
> >>     a log2 scale, or more graphically a bit shifting over a bunch of 0s,
> >>     which is somewhat more resembling to the physical action on the strip.
> > Since I'm on a deadline for this and making changes to this in the
> > kernel would take too long, I don't think I'm going to advocate for
> > changing this behavior right now. Although I do agree that eventually it
> > should be changed. Graphically, a bit moving across a field like that
> > makes sense, but I think that would be a difficult value to make
> > practical use of in an application without changing it to a simple 1-13
> > value. If I get far enough that I can start implementing support for
> > tactile strips and all those other fancy features some of these tablets
> > have I might have it convert the values for tactile strips like that to
> > something more usable by default and leave the other axes as-is. I'm
> > curious to hear Ping and Jason's opinion on this though, and what kind
> > of advantages
> >
> Having the pad functions work well is very useful, but definitely
> secondary to proper pen support. The way our driver (ab)uses the
> kernel's event interface to send wheel data is a minor tragedy, and is
> something I am very much hoping Benjamin and I will be able to address
> in the coming months (I've been side-tracked by other things for the
> past few weeks, but just turned my mind back to the problem). If
> working around what currently exists is too much of a time sink, I
> wouldn't worry too much: pad support is useful, but clearly secondary
> to getting the pen working properly.

As far as the "glitches" are limited to a known bunch, I guess it might
not be much of a problem to have those worked around, might be harder
though if input_absinfo info is forwarded, unless we make it as-is or
not depending on the value.

But I agree this is a secondary goal, if a fix is approachable in
drivers, we can just use the same approach than for every other axis and
wait for the fix there.

> 
> >>   - Another device has one wheel, reported through ABS_WHEEL. Even though
> >>     min/max are reported as [0..1023], on interaction it goes [0..71] (funky
> >>     range too)
> >>
> >>   We could just forward this as-is, but seems hindering enough to do anything
> >>   useful with those unless that behavior is corrected.
> >>
> >>   When supported, IMO it'd make sense to have those axes behave similar to
> >>   scroll axes, so the axis value increments or decrements depending on the
> >>   direction. I'm not sure if there would be cases where the absolute value
> >>   matters here?
> >>
> >> * One thing worth noting is that axes are currently normalized, to [-1..1]
> >>   for stylus tilt, and [0..1] for everything else. I remember Peter's
> >>   tablet wayland protocol proposal basically forwarded input_absinfo, this
> >>   might not be fully compatible with that, although TBH I'm unsure
> >>   clamping/normalization should take place so high in the stack...
> > I'm with Peter on this actually. If the axes were used for something
> > else I might approve of normalization in libinput but I think having
> > absolute values fits more of the use cases for the extra axes found on
> > many tablets, especially since Ping said that some of Wacom's in-house
> > applications actually need these. I do think however, that maybe we
> > should consider clamping axis values with libinput even if we don't
> > normalize the axes by default.
> >
> IMO, normalization makes a lot of sense and libinput is a fine place to do it.
> 
> Applications use axis data for various reasons. For instance an app
> may read pressure values to adjust brush size, change the ink flow
> rate, or possibly even simulate the pressure-varying effects of a
> paint knife. When using data, the application can interpret it in two
> ways*: as unitless quantity or as a number with some attached unit.
> _Many_ applications don't particularly care about units and will
> simply ignore any unit information provided. They take the data,
> normalize it, and use that fractional value to scale some other
> quantity (brush size, flow rate, degree of paint drag). Even data that
> you normally associate with a unit is typically treated this way. For
> instance, tilt angles are liberally interpreted and as long as the
> values are centered and increase/decrease properly most applications
> could care less about the underlying units**. For these applications,
> normalizing the data up-front just makes things easier: the API is
> cleaner since you just move around a bunch of floats, and it saves
> applications from doing the normalization that they're going to do
> anyway.

You make a good point with tilt axes below. I think this at least stands
true for ABS_PRESSURE, a device with 1024 possible values doesn't
register 4x as much pressure as a device with 256 values, those are
rather the "sensibility" of the axis, same to some extent for
wheel/strip axes I'd say.

For the others, knowing the units/resolution can be indeed useful
depending on the context, so the lowest common denominator seems
input_absinfo.

And I should add, IMO it would make sense to relate axes together and be
able to set/apply a transformation matrix on those, it would be nice to
get tilt to match device orientation (eg. left-handed modes).

> 
> Now, that said, there is a small class of applications that make use
> of the units for one reason or another. Perhaps it is a drafting
> application that measures physical distances in cm between two points,
> or perhaps it is that drawing application from earlier now modeling
> the physical effects of a given number of newtons of force on the
> tool/ink/canvas. These kinds of uses are niche, but we should
> absolutely not prevent developers from being able to make them. What
> is important to the developer in this case is that the data is
> accompanied with enough information to put it into physical context;
> at the very least this means transmitting the physical resolution of
> the data. Now, you could provide the raw data to an application but
> there's not really any point. One application might want to work in
> centimeters, the next in inches, the next in meters. In short, though
> applications interested in units aren't going to /normalize/ the data,
> there's a good chance they're going to /rescale/ the data. And if
> they're rescaling the data to their unit-of-choice anyway, it doesn't
> really matter what unit the data is ingested in since the resolution
> information is available.

This certainly looks like a source of bigger pitfalls than
normalization, but from the looks of it, those apps are already into
unit transformation :). For those cases it would certainly help to
document which units does libinput use for each axis.

> 
> Based on that, I think it makes a lot of sense to pre-process the
> data. Applications don't use the raw values as-is anyway, so if
> pre-processing allows for a simpler and easier-to-understand interface
> then it should be seriously considered. Some axes might make more
> sense canonicalized rather than normalized (e.g. angles might make
> more sense in radians rather than fractions of a [semi-]circle) but
> that's a separate issue.
> 
> 
> * There is a third vanishingly small class of applications that
> neither normalize their data nor care about the unit of the data.
> They, for whatever reason, need to know the underlying raw values.
> This could be for something like a hardware debug tool or simulator. I
> don't think this class of applications is really libinput's target. If
> an application really-truly-absolutely needs to know the raw values,
> it should use a low-level interface like evdev.
> 
> ** I'd argue this is the case given that both the Qt and GTK
> frameworks have problems with tilt angles. The former specifies data
> to be in degrees but doesn't convert non-degree data, while the latter
> provides no guarantee of units at all and in practice applications
> like GIMP wind up computing the wrong physical angles.

I can talk for Gimp/GTK+. Effectively axes are normalized in GDK, close
to the windowing layer. Gimp applies through some f(x) to axes, so those
could be theoretically corrected wrt reality.

But I agree this sounds broken, I for one didn't know ABS_TILT_* had
meaningful units :).

> 
> > I've forked libinput and I have a branch where I'm fixing up the patches
> > Carlos sent in based on the feedback from Peter. You can find it here:
> >
> >         https://github.com/Lyude1337/libinput/tree/carlos_cleanup
> >
> > The history is messy on this, but once this is ready to get sent in as
> > actual patches I'll be rebasing the history.
> >
> > Right now I've removed normalization from my branch, but if someone
> > brings up a good reason to actually have libinput handle that then I can
> > revert the change.
> >
> > Another thing I'm considering regarding this is just having libinput
> > provide functions/macros for normalizing the values so applications can
> > normalize the values easily if they want.
> >
> That could be an interesting approach as well. After Qt and GTK both
> screwed up tilt, I don't trust clients as far as I can throw them.
> There should be some way to get normalized/canonical data, whether
> that's the default (which I'd currently argue), or through some
> conversion functions.

I would agree to go that way, just having some helper defines/functions
would suffice here.

Cheers,
  Carlos