[PATCH libinput 0/24] Tablet support

[Jason Gerecke]

On Wed, May 28, 2014 at 6:39 PM, Peter Hutterer
<peter.hutterer at who-t.net> wrote:
> On Wed, May 28, 2014 at 10:23:00AM -0700, Jason Gerecke wrote:
>> On Tue, May 27, 2014 at 8:32 PM, Chandler Paul <thatslyude at gmail.com> wrote:
>> > On Wed, 2014-05-28 at 12:48 +1000, Peter Hutterer wrote:
>> >> On Tue, May 27, 2014 at 03:40:07PM -0700, Jason Gerecke wrote:
>> >> > On Tue, May 27, 2014 at 3:20 PM, Peter Hutterer
>> >> > <peter.hutterer at who-t.net> wrote:
>> >> > > On Tue, May 27, 2014 at 04:32:14PM -0400, Chandler Paul wrote:
>> >> > >> On Tue, 2014-05-27 at 13:11 -0700, Jason Gerecke wrote:
>> >> > >> > I've been away from my computer for most of the (long) weekend up
>> >> > >> > here, so apologies for being a bit quiet :)
>> >> > >>
>> >> > >> > There's a subtlety on the protocol side of things that can't be
>> >> > >> > ignored. When normalizing data, you want to be careful to preserve
>> >> > >> > information about the zero point. Without that, you can't meaningfully
>> >> > >> > pass the data along. Lets imagine that we have some sensor that will
>> >> > >> > report values between 10 and 100, with a resolution of 1 unit = 1
>> >> > >> > elbow per square ounce. If we normalize that to the range [0,
>> >> > >> > UINT32_MAX] we've lost information about where "zero" is. A normalized
>> >> > >> > value of zero does not correspond to zero elbows per square ounce as
>> >> > >> > you might expect, and the resolution info is insufficient to correct
>> >> > >> > the offset.
>> >> > >> >
>> >> > >> > Now, if we've done our jobs properly in libinput, that shouldn't be a
>> >> > >> > problem. We would have normalized that sensor's values to [0.1, 1] and
>> >> > >> > announced the axis to have a resolution of 1 unit = 100 elbows per
>> >> > >> > square ounce. Because the zero point is offset like it originally was,
>> >> > >> > it's preserved through the scaling done for the protocol and so the
>> >> > >> > original 10-100 range can be recovered. The only amendment I'd make is
>> >> > >> > to use a signed integer type rather than an unsigned one, since we may
>> >> > >> > have negative normalized values that need to be sent through the
>> >> > >> > protocol.
>> >> > >> I just wrote code to normalize it to INT_MAX, but since everything's in
>> >> > >> fixed point integers the actual values it's being scaled to are
>> >> > >> 0-8388607.99609375 when the fixed point axis value is converted back
>> >> > >> into a double, which as I'm sure you probably realize is kind of a
>> >> > >> strange value, and I'm starting to think something like 0.1-1.0 would be
>> >> > >> a lot better, trying to normalize to INT_MAX results in something that
>> >> > >> sounds really weird to work with.
>> >> > >
>> >> > > we need a LI_FIXED_MAX then. Normalising to 0-1 in a 24.8 fixed point only
>> >> > > leaves us with 256 value per axis.
>> >> > >
>> >> > Yeah, we don't want to pass a value like that through the fixed type.
>> >> > It either needs to be re-scaled to use the full range (be that
>> >> > [INT_MIN, INT_MAX] or [FIXED_MIN, FIXED_MAX]) or sent with a type that
>> >> > won't loose quite as many bits :D
>> >>
>> >> didn't think of it until after I sent the previous email:
>> >>
>> >> there's a side-effect that we need to be aware of: if we scale an axis to
>> >> LI_FIXED_MAX, we're effectively guaranteed to get 32-bit integer overflows
>> >> on operations with that value. So we're effectively forcing the caller to
>> >> work with int64_t to be on the safe side.
>> >>
>> >> which isn't the worst of all things: on fixed 24.8 with current devices
>> >> overflows aren't _that_ hard to trigger so we pretty much get to pick
>> >> whether they happen sometimes for some devices or all the time, effectively
>> >> forcing all callers to handle this correctly from day 1.
>>
>> I thought that scaling-up the range to LI_FIXED_MAX was purely an
>> intermediate form that would then be scaled-back to either a
>> normalized or canonical value by the wl-client. I don't know who in
>> their right mind would try to use the full-range 24.8 data directly
>> without first dividing it down.
>
> remember, libinput output != what the wl client sees. the rough architecture
> is:
>   kernel -> libinput -> compositor |wl protocol| wl toolkit - application
>
> so the compositor decides what's on the protocol and libinput merely adds as
> a filter. the compositor can't add to what libinput provides, but it can
> filter or modify values it gets from libinput.
> or to compare it to the Xorg stack, think of libinput as the equivalent to
> the wacom/evdev/synaptics drivers - they generate events but don't have a
> say in what the clients eventually see.
>
> so if libinput provides scaled output that may or may not be what the
> clients see, but that depends on the wl protocol. having said that, I'd
> probably say the wl protocol should use 24.8 normalized as well, so what you
> say still applies :)
>
I completely agree, but until I did a little more reading could not
see how that had any bearing on what I said. Now though, I think I
might see a missing puzzle piece.

I was under the impression that libinput would output values as a
bog-standard 'float' or 'double' type with range [-1, 1]. The caller
(e.g. Weston) could do whatever they wanted with these values, and
then send them across the Wayland protocol to some client. Because
Wayland does not have a suitable type to represent values like that,
the values would first need to be re-scaled for transmission (e.g. as
a 'wl_fixed_t') and then immediately scaled back down by the client
after receiving them. There's no concern about overflow because the
scaled form is just an intermediate representation.

Let me know if I'm wrong, but it appears that you advocate exposing a
full-range 'li_fixed_t' (rather than a floating-point type) to the
callers. That would mesh with your overflow concerns, and makes sense
in the context of what you've been writing...

If that's the case though, I have to ask -- *why* on Earth is
'li_fixed_t' even being considered as libinput's output type? I know
I'm missing something because the thought seems like such a blindingly
terrible choice. Its not an intrinsic type, is ill-suited to the task
of representing normalized data, and can't be directly used without
very real overflow concerns. By contrast 'float' is a standard type
which takes up the same space, offers sufficient precision of
normalized data, and can be freely operated on without concern. Is
there some crazy reason that floats aren't viable as the output type?
Are we communicating over the Wayland protocol despite being a
library? Targeting hardware with no FPU? libinput seems like it could
be useful to more than just Wayland compositors, so I _really_ hope
the reason isn't to maximize the degree of integration between the
two.

>> > I've already implemented scaling to LI_FIXED_MAX in my tree, but I agree
>> > this is something we should probably try to avoid. My thought here is
>> > that we could scale it to a smaller value that's extremely unlikely to
>> > ever overflow. My idea is to scale it to a 16.8 bit value. I don't think
>> > it's likely that we'll ever be encountering anything larger then a 16
>> > bit value with a wacom tablet. This does kind of seem like a bit of a
>> > hack though.
>>
>> There are actually a handful of tablets that exceed 16 bits for their
>> X/Y axes. The Cintiq 24 for instance has a logical resolution of
>> 104480x65600.
>>
>> >>
>> >> > >> Also, what exactly is a "zero-point" in this context?
>> >> > >
>> >> > > whatever the neutral state of an axis is. e.g. tilt goes in both
>> >> > > directions so the effective range is -value ... 0 ... +value.
>> >> > >
>> >> > Or as I like to think about it, the point where an axis would report a
>> >> > value of "0". Most axes will report that value, and you'll just want
>> >> > to be sure that an input of 0 turns into an output of 0. There's a
>> >> > corner case where 0 could be outside the range of the axis, but if you
>> >> > treat it as though it could you'll end up mapping the range
>> >> > appropriately (e.g. treating an axis that only reports [10, 100] as
>> >> > though it really reported [0, 100] will result in proper
>> >> > normalization).
>> >>
>> >> new question: what about a device that has an zero state that's not halfway
>> >> between min/max? currently it'd advertise a range of [-N, M]. normalising
>> >> those means we lose that information.
>> > I don't think there's any devices like that right now, but if there are
>> > it might be better just to pass the value to the client as-is without
>> > normalizing it like we do with the pressure axis.
>> >>
>>
>> I'm confused. I don't expect axes exposed through libinput to
>> advertise a range. The range is implicitly [-1, 1] because its a
>> normalized value. Some hardware may never report certain values (e.g.
>> an Intuos currently won't report a negative X value) but the average
>> software should not particularly care about that fact.
>
> [...]
>
>>
>> The evdev device reports canonical values in the range [-100, 511]. A
>> resolution of 1000 units/kilogram-force has been set, allowing
>> applications that care about absolute forces to recover the original
>> value. When libinput adds this device, it normalizes the device range
>> to [-1, 1] being sure to map an input value of 0 to an output value of
>> 0. Because the input range is asymmetric, some values in its implicit
>> [-1, 1] range will never be sent. The Wayland protocol does not have a
>> convenient "float" type that can accurately contain these values, so
>> it scales the implicit input range to match the full range of some
>> type (e.g. "fixed" and its [-8388608, 8388607.99609375] range). On the
>> receiving end, GDK uses wl-client to scale the full-range value back
>> to normalized and then clamps it to the range [0, 1] since that is the
>> documented range of GDK_AXIS_PRESSURE. GIMP is notified by GDK of the
>> pressure and uses it as a scaling factor on the brush size to achieve
>> some size between zero and 8 pixels.
>
> yeah, I guess that's what I'm not a big fan of: reporting a range that the
> device cannot actually achieve. If you switch your example around and you
> have a device that goes from [-511, 100], GIMP will never scale up to the 8
> pixels, despite the user using the maximum pressure on the device.
>
> Which may not matter after all, maybe no-one cares about this anyway. It
> would be easy enough to add at a later point to say that effectively,
> min/max is bounded by $whatever, not by the normalized range.
> but at that point we're now providing exactly what the kernel gives us, but
> mangled into a different format.
>

I suppose its a matter of interpretation. I don't see much of a
problem with the case you list above (practically speaking, I'll
notice the brush dynamics don't suit me and adjust GIMP's settings...)
but I also won't argue that the range information is useless since I
can imagine limited cases where knowing that you've bottomed-out could
be useful (though again, practically speaking, its should be pretty
obvious when either the tool or its in-computer representation stops
responding to "more" input ;))

>> GIMP (and the majority of desktop applications) do not care about the
>> absolute range of the input data because they only use the normalized
>> value as a scaling factor. If we made a slightly different assumption
>> about how our device above works (lets say it has a range of [100,
>> 511] instead of [-100, 511]) it doesn't change the picture for GIMP:
>> 50% of the maximum force will normalize to a value of 0.5 either way,
>> and will result in a brush 4px in diameter. Its possible that an
>> application which pays attention to units could make use of range
>> information, but only insofar as letting the user know that some
>> values are beyond the sensor's capabilities.
>
> ..and we're also assuming that 0 actually means zero/neutral on a device,
> and devices never have a truly arbitray axis range of [-N, M] with the
> neutral point at 0 + X.
>
> Cheers,
>    Peter

That is an assumption, yes. Though I'd argue such a device has a buggy
kernel driver (or HID descriptor). Our own ABS_TILT_{X,Y} axes are an
example: we currently report a range of [0, 127] and have a zero-point
at 64 :/ The X driver (as of 0.18.0) contains a patch that should let
us switch that to [-63, 64] without breaking, and I'm planning on
doing so in the future.

Jason
---
Now instead of four in the eights place /
you’ve got three, ‘Cause you added one  /
(That is to say, eight) to the two,     /
But you can’t take seven from three,    /
So you look at the sixty-fours....