Comments about adding tablet support to Wayland

Ping Cheng pinglinux at gmail.com
Tue Jul 1 12:26:03 PDT 2014 

On Tue, Jul 1, 2014 at 10:47 AM, Ping Cheng <pinglinux at gmail.com> wrote:
> On Mon, Jun 30, 2014 at 1:03 AM, Dmitry Kazakov <dimula73 at gmail.com> wrote:
>> Hi, all!
>>
>> I'd like to add about mapping of the tablet input.
>>
>> In XInput one can assign a matrix transfromation for each tablet device,
>> which is exactly what people need.
>
> If we support matrix transformation in libinput/Wayland, all "tablet
> to screen" mapping requests would be met, including my mapping to
> windows.
>
>> The usual usecase is the following:
>>
>> A painter has a display + intuos tablet + cintiq. The external tablet is
>> mapped to the external display, the Cintiq, obviously, is mapped into itself
>> [0]. The external display can be (and probably will be) rotated on 90deg to
>> have Portrait orientation.
>>
>> Right now all the mappings for such configuration can be achieved with
>> xsetwacom/xinput calls.
>>
>>
>> [0] - No painters I know use mappings to the specific window.
>
> You work with painters directly. I believe you know more of what they
> use. I am only a driver developer. I can not draw a straight line on
> paper or on computer ;).
>
>> Mode switches are troublesome.
>
> My mapping/display toggle/mode toggle experience was from animation
> artists' requests. What kind of trouble do you usually see from mode
> switch/toggle? I want to see if we have a solution.

>From discussions in other threads, I feel my terms (mapping, display
toggle, mode toggle) may not clear to some of you.

Mapping means to map a specific tablet area to a specific display
area. A common use case would be to map whole tablet to a screen/
whole desktop;

Mode toggle is a function to switch between absolute to relative motion mode;

Display toggle is a function to switch among screens and the whole
desktop, in multi-monitor setup.

Ping

> Thank you for sharing your ideas. We desperately needed feedback like yours.
>
> Ping
>
>> On Mon, Jun 30, 2014 at 11:41 AM, Dmitry Kazakov <dimula73 at gmail.com> wrote:
>>>
>>> Hi, all!
>>>
>>> I will try to answer all your emails in one post.
>>>
>>> 1) Tablet vs. Touch. Basically what I am talking about is *not* touch
>>> arbitration and palm recognition. The Wacom driver itself blocks all the
>>> touch events when the stylus is in the proximity, so it performs the
>>> necessary arbitration. What we need is to block several kinds of actions
>>> (read "tools") when the user uses touch and unblock them when the user works
>>> with stylus or mouse. An exact usecase: we need to allow the user to
>>> zoom/pan on the canvas with his fingers, but at the same time real painting
>>> with fingers must be prohibited (it disturbs the painters). Right now it is
>>> almost impossible, because we get no touch events, and instead get
>>> synthesized MousePress/Release/Move events for finger actions, which are
>>> indistinguishable from real mouse events.
>>>
>>> 2) Buttons vs Tablet Press. Well, to tell you the truth we are not
>>> interested in whether the stylus touched the surface of not. At all. The
>>> stylus has 3 button: the tip itself, and two buttons. These three buttons
>>> can be remapped with the driver (xsetwacom) to any X11 button the user
>>> wants. Therefore, I have absolutely no interest in TabletTouch/TabletRelease
>>> if the only thing they tell me that the stylus has touched the surface of
>>> the tablet with the stylus or not. What I really need is the
>>> TabletButtonPress/Release event, which tells me:
>>>
>>> 1) Which button (of the three) is pressed
>>> 2) Exact position of the stylus
>>> 3) Pressure, Tilt, Z-coordinate if available (might be zero/unit for some
>>> of the buttons)
>>>
>>> The situation is getting even worse if you look at the feature which
>>> Windows' Wacom driver has (I'm not sure whether this feature is available in
>>> X11 Wacom driver, but it is highly requested by the painters). On Windows
>>> the buttons on the stylus can be switched into "modifier" mode. That is
>>> pressing the button doesn't produce a real button event. You need to press
>>> the  stylus button, and then touch the surface of the tablet with the tip:
>>> only then the app will get mouse button click (right or middle button
>>> usually). If this feature will ever be implemented in X11 Wacom driver
>>> (which is quite desirable), your protocol with TabletTouch/Press will not
>>> work. Please check Wintab protocol docs for more info, specifically,
>>> CSR_SYSBTNMAP attributes [0]
>>>
>>> 3) Axes resolution. Yes, it is perfectly ok to have a separate function
>>> which tells the physical limits of the axis. What I wanted to say is that
>>> min_value/max_value attributes, which are reported by XInput are not enough.
>>> For rotation I also need to know the mapping of the coordinate system origin
>>> and it's direction (clockwise/counterclockwise).
>>>
>>> PS:
>>> Please keep me in CC, I'm having troubles with keeping up with the traffic
>>> in this mailing list.
>>>
>>>
>>> [0] - http://www.wacomeng.com/windows/docs/Wintab_v140.htm
>>>
>>>
>>>
>>> On Fri, Jun 27, 2014 at 6:53 AM, Jason Gerecke <killertofu at gmail.com>
>>> wrote:
>>>>
>>>> On Wed, Jun 25, 2014 at 5:50 PM, Peter Hutterer
>>>> <peter.hutterer at who-t.net> wrote:
>>>> > Replying to three emails at once here to keep the thread a bit more
>>>> > managable.
>>>> >
>>>> > On Wed, Jun 25, 2014 at 01:38:22PM -0700, Jason Gerecke wrote:
>>>> >> On Wed, Jun 25, 2014 at 12:38 AM, Lyude <thatslyude at gmail.com> wrote:
>>>> >> > On Wed, 2014-06-25 at 11:06 +0400, Dmitry Kazakov wrote:
>>>> >> >> Hi, all!
>>>> >> >>
>>>> >> >>
>>>> >> >> I am a developer from Krita painting application team. We recently
>>>> >> >> did
>>>> >> >> quite much work on incorporating better tablet support in Krita. I
>>>> >> >> have several comments about your proposal of the tablet protocol
>>>> >> >> (sorry for nor replying directly, since I wasn't subscribed to the
>>>> >> >> list before).
>>>> >> >
>>>> >> > Benjamin commented on this as I was writing this e-mail, and I
>>>> >> > figured I
>>>> >> > should too: yes, it's awesome to see developers commenting on the
>>>> >> > protocol. A lot of the quirks around this protocol are going to be
>>>> >> > difficult to see without the help of people who have programmed on
>>>> >> > the
>>>> >> > client side of things as opposed to the compositor side of things.
>>>> >> > So
>>>> >> > yes, your input is very much appreciated and I thank you for it!
>>>> >> >
>>>> >> >>
>>>> >> >> 1) Axes. There should also be an axis for rotation of the stylus
>>>> >> >> (Artpen) and Tangential Pressure (for the wheel of the Airbrush).
>>>> >> >>
>>>> >> >> 2) There is also Artpen type of stylus. In Qt it is called
>>>> >> >> "Rotational
>>>> >> >> Stylus".
>>>> >> >
>>>> >> > Don't worry, we haven't forgotten about these. These will eventually
>>>> >> > be
>>>> >> > added into the protocol. The reason why they're not in this draft is
>>>> >> > because I'm doing this as my Google Summer of Code project, and as
>>>> >> > such
>>>> >> > I'm on a deadline and I have to focus on just getting the basics
>>>> >> > done
>>>> >> > first before I can focus on all of the other features.
>>>> >
>>>> > Our current approach, both in libinput and the WL protocol should make
>>>> > these
>>>> > additions little more than adding a couple of enum, so I think we're
>>>> > good
>>>> > here.
>>>> >
>>>> >> >> 3) Fingers. There is a complication in XInput2 right now, since
>>>> >> >> touch
>>>> >> >> enabled Wacom devices have a special Finger XInput2 device, which
>>>> >> >> provides both interfaces: tablet and touch and therefore generates
>>>> >> >> both types of events. Right now Qt5 still cannot handle it
>>>> >> >> properly,
>>>> >> >> but the work is in progress. From Krita point of view, the main
>>>> >> >> usecase for us is to distinguish whether the user paints with a
>>>> >> >> finger
>>>> >> >> of with the stylus. Because most of the users prefer to disable
>>>> >> >> painting with fingers and use it for gestures/UI only (yes, palm
>>>> >> >> detection works with non-100% probability).
>>>> >> >
>>>> >> > Right now libinput handles the finger device as another touchpad,
>>>> >> > since
>>>> >> > that's usually what it is. Your use-case sounds perfectly valid
>>>> >> > though,
>>>> >> > but IMO a better approach would be to add something to the protocol
>>>> >> > for
>>>> >> > touchpads on wayland so that it can be known that they belong to a
>>>> >> > tablet and provide any other sorts of data you might be need, so
>>>> >> > programs like yours can treat them differently.
>>>> >
>>>> > Dmitry, are you talking about pen/touch arbitration, i.e. don't send
>>>> > touch
>>>> > events when the pen is in use. If so, that's definitely on the plan, we
>>>> > need
>>>> > it for touchpads (disable while typing feature), and we need it for the
>>>> > pen/touch interference.
>>>> >
>>>> > This will be hidden away so you or event the compositor don't have to
>>>> > worry
>>>> > about it.
>>>> >
>>>> >> >> 4) Button Press/Release events should come in both cases: when the
>>>> >> >> user clicks on the stylus' buttons and when the stylus touches the
>>>> >> >> surface of the tablet.
>>>> >> >
>>>> >> > I'm not entirely sure that's a good idea. If I'm reading this right,
>>>> >> > you
>>>> >> > mean that additional button presses should be sent when the tool
>>>> >> > touches
>>>> >> > the surface of the tablet. [...]
>>>> >
>>>> > We're already sending out BTN_TOUCH when the tip touches the surface,
>>>> > so I
>>>> > think we're good here. Unless Dmitry was referring to something else.
>>>> >
>>>> >
>>>> >> >> 6) It might be a good idea to define the physical properties of the
>>>> >> >> axes. E.g. for tilt, rotation and tangential pressure. Afair, Wacom
>>>> >> >> driver for XInput returns some not-very-obvious values right now.
>>>> >> >> One
>>>> >> >> would need to experiment to know what these numbers mean.
>>>> >> >
>>>> >> > We would all love for this to be the case I promise you, but
>>>> >> > unfortunately it's not that simple for all of the axes. The distance
>>>> >> > axis reports a seemingly meaningless value that can't be converted
>>>> >> > to
>>>> >> > millimeters very easily. That being said, I have come up with a few
>>>> >> > ways
>>>> >> > that we could actually convert it to millimeters, but this will have
>>>> >> > to
>>>> >> > wait until I've fulfilled the goals for my Google Summer of Code
>>>> >> > Project
>>>> >> > (unless anyone else wants to implement this in the mean time in
>>>> >> > libinput
>>>> >> > of course).
>>>> >> >
>>>> >> > I'll write up the method I've come up with for converting this wild
>>>> >> > value to actual millimeters at some point when I get the chance.
>>>> >> >
>>>> >> While I'm interested in seeing what you've come up with, I would be
>>>> >> very hesitant to integrate the code into libinput. We make *no* claims
>>>> >> about the physical resolution or accuracy of the distance axis. I've
>>>> >> seen the value change by more than 10 units just by switching to a
>>>> >> different pen... There's absolutely nothing stopping us from
>>>> >> introducing a tablet that invalidates any clever code you may come up
>>>> >> with.
>>>> >
>>>> > yeah, I agree with Jason here, let's not pretend we have data we don't
>>>> > have.
>>>> > As much as I'd like to attach concrete physical information to all axes
>>>> > we
>>>> > can't (well, shouldn't) make it up.
>>>> >
>>>> > Short of keeping a database of each tablet with the offsets and ratios
>>>> > to
>>>> > convert distance values to mm I don't think this is doable, and I'm not
>>>> > a
>>>> > big fan of that either.
>>>> >
>>>> >> > As for the tilt axes of the tools, this is something that could be
>>>> >> > represented in a more meaningful value. We could normalize it to a
>>>> >> > number between 0 and 180°, so you can get the actual tilt in degrees
>>>> >> > as
>>>> >> > opposed to what we currently have. This is something we could pull
>>>> >> > off
>>>> >> > rather easily, I'll make sure to discuss this with my mentor
>>>> >> > tomorrow.
>>>> >> >
>>>> >> Alternatively, I would suggest adding an e.g.
>>>> >> `libinput_event_tablet_get_axis_resolution` function. To get a
>>>> >> physical value, a caller would just multiply whatever this function
>>>> >> returns for some axis by the current normalized unitless value for the
>>>> >> same. I suggest this for a few reasons: it is a fairly standard way of
>>>> >> doing things (HID, the kernel, and X all use the same system), is more
>>>> >> efficient (callers working in your preferred unit [e.g. Qt] are spared
>>>> >> doing the conversion but anyone else [e.g. GTK, EFL] will have to do a
>>>> >> second conversion to a different unit), and provides a way of getting
>>>> >> physical information for any axis (if we ever came out with a pen that
>>>> >> accurately measured distance, you wouldn't need to change the
>>>> >> semantics of how distance is reported or sacrifice the now-existant
>>>> >> physical translation information).
>>>> >
>>>> > Jason: how accurate is tilt, and are applications actually using it as
>>>> > angle or
>>>> > just as normalized number anyway?
>>>> >
>>>>
>>>> I can't find a mention of the accuracy in any public docs, so I'm a
>>>> little hesitant to give numbers on the list. The spec sheet does show
>>>> it to be accurate to a handful of degrees though.
>>>>
>>>> The second part is a little complicated to answer. Every tilt-enabled
>>>> program I'm aware of uses the data to adjust the brush azimuth. The
>>>> goal is to have the virtual brush angled to be parallel with the
>>>> physical pen at all times. A few applications also calculate an
>>>> altitude angle, deforming the brush from circular to increasingly
>>>> eliptical as the pen becomes more horizontal. Qt and Android (and
>>>> likely EFL based on current discussions) have APIs that specify angles
>>>> in various physical forms: tilt-x/tilt-y in degrees, alt-az in
>>>> radians, etc. GTK on the other hand provides normalized data, but
>>>> neither provides resolution information nor information about what
>>>> "normalized" means. Because of the resulting ambiguity, GIMP and
>>>> Inkscape do their calculations on the assumption that [-1, 1] in GTK
>>>> corresponds to [-180 degrees, +180 degrees] in the physical world. GTK
>>>> actually does its normalization based on the device's min/max though
>>>> (so [-64 degrees, +63 degrees] for our hardware) meaning that GIMP and
>>>> Inkscape wind up calculating incorrect azimuth values. The results
>>>> aren't _that_ wrong though; I'm not aware of anyone having noticed or
>>>> filing a bug about it.
>>>>
>>>> tl;dr, Applications universally /try/ to use the physical angles. Not
>>>> all succeed.
>>>>
>>>> Jason
>>>> ---
>>>> Now instead of four in the eights place /
>>>> you’ve got three, ‘Cause you added one  /
>>>> (That is to say, eight) to the two,     /
>>>> But you can’t take seven from three,    /
>>>> So you look at the sixty-fours....
>>>>
>>>> >> > As for tangential pressure, this is a term I've never actually heard
>>>> >> > of
>>>> >> > before. I don't know what the values from the tablet are supposed to
>>>> >> > correspond to in regards to pressure, so I've added two of our
>>>> >> > friends
>>>> >> > at Wacom to the CC list help us out (I hope you two don't mind!) on
>>>> >> > this.
>>>> >> > (Jason and Ping, if you guys aren't on the list already, the
>>>> >> > original
>>>> >> > protocol this e-mail is discussing can be found here:
>>>> >> >
>>>> >> > http://lists.freedesktop.org/archives/wayland-devel/2014-June/015583.html
>>>> >> > )
>>>> >> >
>>>> >> > In regards to rotation and any other axes, I haven't had any contact
>>>> >> > with these yet. So I can't really say much on them.
>>>> >
>>>> > Rotation exists on the mouse/lens cursor tool, but it's a calculation
>>>> > based
>>>> > on the tilt x/y axes. For us it'd just be one more axis.
>>>> >
>>>> > Cheers,
>>>> >    Peter
>>>> >
>>>> >> I'm not sure where the term "tangential pressure" came from, but it
>>>> >> can refer either to the fingerwheel on the airbrush tool (expected to
>>>> >> be used to control ink flow rate; value is basically [0, 1]) or the
>>>> >> fingerwheel on the 4D Mouse tool (which is like a spring-loaded
>>>> >> mousewheel that reports how far forward or backward from the neutral
>>>> >> position the wheel is; value is basically [-1, 1]).
>>>> >>
>>>> >> Jason
>>>> >
>>>
>>>
>>>
>>>
>>> --
>>> Dmitry Kazakov
>>
>>
>>
>>
>> --
>> Dmitry Kazakov

More information about the wayland-devel mailing list