[RFC DRAFT] graphics tablet protocol extension

[Peter Hutterer]

On Wed, Oct 02, 2013 at 05:44:29PM +0200, David Herrmann wrote:
> Hi Peter
> 
> On Fri, Sep 20, 2013 at 12:35 PM, Peter Hutterer
> <peter.hutterer at who-t.net> wrote:
> > I've been working on a protocol extension to support graphics tablets such
> > as the Wacom set of tablets, and I'm now at the stage where I'd like a few
> > comments. I was hoping that I'd get a full implementation before XDC but
> > unfortunately that didn't happen, so for now I'll just show the protocol.
> > I've got a PoC implementation, but it's missing a few too many pieces to be
> > actually usable just yet.
> >
> > Any feedback appreciated. This is a relatively early stage, so there are
> > still many changes expected.
> >
> > The goal is to make it possible to access graphics tablets. One important
> > thing to note is that this interface does _not_ cover the touch ability of
> > some tablets. This should go through wl_touch (for touchscreen-like tablets)
> > or wl_pointer (for external touchpad-like tablets).
> >
> > There are a few notable differences to the wl_pointer interface:
> > * tablets have a tool type that matters, it lets applications such as the
> >   GIMP select paint tools based on the physical tool. those tools also often
> >   have HW serial numbers to uniquely identify them.
> > * extra axes mattter: pressure, tilt, distance - all influence the stylus
> >   behaviour
> > * more than one device may be present, so it's important to have access to
> >   all devices on a one-by-one basis. unlike wl_pointer, where we just have
> >   one virtual pointer.
> > * proximity matters since we can leave proximity from directly above a
> >   surface. the pointer can't do that, it moves from one surface to the next.
> >   so in some ways it's closer to wl_touch in that regard.
> >
> > Some design notes:
> > * generally most axes change at the same time, hence the choice to send a
> >   wl_array instead of separate events.
> > * x/y would have to be adjusted relative to the surface, but technically the
> >   same would have to be done to e.g. distance on a true 3D desktop.
> > * not sure at all about the relative events at all or if there's a need for
> >   it. iirc only some styly have REL_*WHEEL, do we need something else? Ping,
> >   Jason?
> > * I don't have a specific touch event, I figured BTN_TOUCH would do the job.
> > * focus handling for the stylus is easy. focus handling for the buttons on
> >   the pad isn't. they could technically be focused elsewhere, like a
> >   keyboard focus. some buttons are definitely stylus based (BTN_STYLUS,
> >   BTN_STYLUS2, etc.) so should go where the stylus is. Should look at what
> >   Win/OSX do here.
> > * bind/binding/unbind - this is like windowed mode in GIMP. do we still need
> >   this? who's actually using this instead of a full-screen app?
> > * tablet_manager is a bit meh, but the only alternative would be to have a
> >   wl_seat::get_tablets request and a wl_seat::tablet_added event. Possible,
> >   but doesn't look that much nicer. but it does away with the indirection.
> >   (read the diff below to understand what I mean here)
> > * if we stick with the tablet_manager, do we need a
> >   wl_tablet_manager::get_tablets request in case the client releases a
> >   tablet it needs again later? or do we expect to re-bind to
> >   wl_tablet_manager?
> > * fuzz/flat should be dropped, I just haven't yet.
> > * I'd really like to enscribe in the protocol that ABS_PRESSURE means just
> >   that and damn anyone else who wants to use 0x18 as axis code. weston
> >   does this for wl_pointer::button, but it's not actually documented. what's
> >   the deal here?
> > * does this even make sense as wl_tablet or should I try first as
> >   experimental weston-tablet interface that then (maybe) moves later to
> >   wayland proper.
> >
> > That's it so far, again, any feedback appreciated. diff below.
> >
> > Cheers,
> >    Peter
> >
> >
> > diff --git a/protocol/wayland.xml b/protocol/wayland.xml
> > index aeb0412..8d10746 100644
> > --- a/protocol/wayland.xml
> > +++ b/protocol/wayland.xml
> > @@ -1235,7 +1235,7 @@
> >      </request>
> >     </interface>
> >
> > -  <interface name="wl_seat" version="3">
> > +  <interface name="wl_seat" version="4">
> >      <description summary="group of input devices">
> >        A seat is a group of keyboards, pointer and touch devices. This
> >        object is published as a global during start up, or when such a
> > @@ -1251,6 +1251,7 @@
> >        <entry name="pointer" value="1" summary="The seat has pointer devices"/>
> >        <entry name="keyboard" value="2" summary="The seat has one or more keyboards"/>
> >        <entry name="touch" value="4" summary="The seat has touch devices"/>
> > +      <entry name="tablets" value="8" summary="The seat has one or more graphics tablet devices" since="4"/>
> 
> What's actually the reason to allow multiple graphics-tablets per
> seat? I thought wl_seat objects respresent a single user interacting
> with your desktop. So if you have two tablets, why not force them to
> be in two different wl_seat objects? Is there ever a reason to have
> multiple tablets in a single seat? What would the use-case be?

my laptop has a built-in wacom tablet (lenovo x220t), but I also have an
Intuos4 plugged in. depending on the use-case I use either, but they should
be in the same seat. two wl_seat objects means two different keyboard foci
(only one of which actually works if you have only one keyboard), etc.
I also know of professional setups that require two tablets or more.

ideally there'd be just one generic wl_tablet per seat, same as wl_pointer
but the tablets differ too much to make that useable. Hence the multiple
wl_tablet objects.

> We could even go further and put tablets into their own seats. Always.
> A pointer and keyboard may be used by a single user at the same time.
> But for a graphics-tablet that doesn't sound like a legitimate
> use-case, does it? But maybe I just have a different view of wl_seat..
> don't know. I somehow have the feeling we never really agreed on how
> to define wl_seat objects. Currently they are just user-defined groups
> of devices with some specific policies ("only one keyboard-object per
> seat").

my understanding of wl_seat is "one group of input devices". that would
usually map to one user. it's already not ideal for the bimanual input but
then again that's even more of a corner case than multiple seats.

once you start moving devices out into other seats, you're in trouble. a
seat defines keyboard events, so anything that has an interaction with the
keyboard can't easily be unpaired. a tablet is a fancy pointer device,
users expect to click somewhere and have it work as a mouse, i.e.
to move the keyboard focus to wherever you just clicked.
moving it into a different seat means you'd have to switch to a mouse to
change the keyboard focus. or, if you have two wl_seats you now need to pair
them somehow to have the right focus behaviour. and that is going to be 
horrible.

> >      </enum>
> >
> >      <event name="capabilities">
> > @@ -1306,6 +1307,19 @@
> >        <arg name="name" type="string"/>
> >      </event>
> >
> > +    <!-- Version 4 additions -->
> > +    <request name="get_tablet_manager" since="4">
> > +      <description summary="return tablet manager object">
> > +        The ID provided will be initialized to the wl_tablet_manager
> > +        interface for this seat. This can then be used to retrieve the
> > +        objects representing the actual tablet devices.
> > +
> > +        This request only takes effect if the seat has the tablets
> > +        capability.
> > +      </description>
> > +      <arg name="id" type="new_id" interface="wl_tablet_manager"/>
> > +    </request>
> > +
> >    </interface>
> >
> >    <interface name="wl_pointer" version="3">
> > @@ -1617,6 +1631,223 @@
> >      </event>
> >    </interface>
> >
> > +  <interface name="wl_tablet_manager" version="1">
> > +    <description summary="controller object for graphic tablet devices">
> > +      A tablet manager object provides requests to access the graphics
> > +      tablets available on this system.
> > +    </description>
> > +
> > +    <enum name="tablet_type">
> > +      <description summary="tablet type">
> > +        Describes the type of tablet.
> > +      </description>
> > +      <entry name="external" value="0" summary="The tablet is an external tablet, such as an Intuos" />
> > +      <entry name="internal" value="1" summary="The tablet is an built-in tablet, usually in a laptop" />
> > +      <entry name="display" value ="2" summary="The tablet is a display tablet, such as a Cintiq" />
> > +    </enum>
> > +
> > +    <event name="device_added">
> > +      <description summary="new device notification"/>
> > +      <arg name="id" type="new_id" interface="wl_tablet" summary="the newly added graphics tablet " />
> > +      <arg name="name" type="string" summary="the device name"/>
> > +      <arg name="vid" type="uint" summary="vendor id"/>
> > +      <arg name="pid" type="uint" summary="product id"/>
> > +      <arg name="type" type="uint" />
> 
> So tablets are server-side created objects? I think we tried to avoid
> that. Only wl_display uses this now, afaik. Even for globals we added
> the wl_registry interface which just announces unique names which the
> client then binds to via wl_registry.bind. So why not turn
> wl_tablet_manager into wl_tablet_registry and provide the same
> functions as the global registry?

two things:
having thought about this (and talked to krh and daniels at XDC) the tablet
manager shouldn't be necessary anymore. It used to have the device_removed
event, but now it's just an intermediate layer that can be removed by moving
the device_added event up into the seat and sending it conditional on either
get_tablets or on binding to the seat with the correct version.


as for moving tablets over to a global: my first attempt was to have
wl_tablet_manager as a global. the server-side objects stay in that case, if
you bind to the manager it sends you events about the tablets. so for three
tablets and the client wanting tablet 2 it could look like that:

  wl_tablet_manager::bind
       event:device added(1)
       event:device added(2)
       event:device added(3)
       request:get_device(some client id, 2)

at this point you still need some sort of ID to identify the tablet when the
client request comes, so you might as well use server-allocated IDs and let
the client discard what it doesn't want:

  wl_tablet_manager::bind
       event:device added(server id 1)
       event:device added(server id 2)
       event:device added(server id 3)
       request:release (server id 1)
       request:release (server id 3)

so I don't know how to get rid of server-allocated IDs. 

as said above, the tablet_manager doesn't really provide that much
functionality at this point, so another alternative is to have wl_tablet
directly as global, but I don't know how to hook that up with seats
then. and you'd still have server-allocated IDs.

> > +    </event>
> > +
> > +  </interface>
> > +
> > +  <interface name="wl_tablet" version="1">
> > +    <description summary="a graphics tablet device">
> > +      This interface describes a graphics tablet, such as Wacom's Intuos or
> > +      Cintiq ranges.
> > +
> > +      Note, this interface does not handle touch input on touch-capable
> > +      tablets. Depending on the tablet, touch input is sent through the
> > +      wl_pointer (if used as a touchpad) or wl_touch interface (if used as a
> > +      touch screen).
> > +    </description>
> > +    <enum name="axis_type">
> > +      <description summary="axis type"/>
> > +      <entry name="relative" value="2" summary="relative axis"/>
> > +      <entry name="absolute" value="3" summary="relative axis"/>
> > +    </enum>
> > +
> > +    <request name="describe">
> > +      <description summary="request axis information">
> > +        Request axis information from a device. Whe a client sends a
> > +        wl_tablet.describe request, the compositor will emit a
> > +        wl_tablet.axis_capability event for each axis on the device.
> > +        To mark the end of the burst of events, the client can use the
> > +        wl_display.sync request immediately after calling wl_tablet.describe.
> > +      </description>
> 
> Why not always send the axis information? I mean once the client
> requests a tablet_manager doesn't it always want to retrieve a
> description? A wl_display.sync directly after receiving device_added
> would act as the barrier. The only reason not to send this information
> would be to safe traffic if multiple tablets are connected and the
> client wants only one of them. Does this really justify the additional
> method? I have no idea how big these data-sets really are..

tablets atm have a couple of buttons and a couple of axes, but nothing
major (iirc it's 10 or less, depending on the tablet). the reason for the
separate request is that the device_added event has some device information
(notably PID/VID and type), so that clients can simply ignore the ones they
don't want without having to deal with axis description events as well.


> > +    </request>
> > +
> > +    <request name="release" type="destructor">
> > +      <description summary="release the wl_tablet object"/>
> > +    </request>
> > +
> > +
> 
> Nitpick: two blank lines
> 
> > +    <event name="removed">
> > +      <description summary="the device has been removed"/>
> > +    </event>
> > +
> > +    <event name="axis_capability">
> > +      <description summary="axis description">
> > +        For relative devices, min and max are 0. A resolution, fuzz, or flat of 0
> > +        denotes an unknown resolution, fuzz, or flat, respectively.
> > +
> > +        Note that while the min/max values are announced by the device, the value is
> > +        not clamped to the actual range by the compositor. A device may send values
> > +        outside this range.
> > +      </description>
> > +      <arg name="type" type="uint" summary="axis type (relative, absolute)" />
> > +      <arg name="axis" type="uint" summary="axis identifier" />
> > +      <arg name="min" type="int" summary="minimum axis value, inclusive"/>
> > +      <arg name="max" type="int" summary="maximum axis value, inclusive"/>
> > +      <arg name="resolution" type="uint" summary="resolution in units/mm"/>
> > +      <arg name="fuzz" type="uint"/>
> > +      <arg name="flat" type="uint"/>
> > +    </event>
> > +
> > +    <event name="button_capability">
> > +      <description summary="button description">
> > +      </description>
> > +      <arg name="buttons" type="array" />
> > +    </event>
> > +
> > +    <enum name="tool_type">
> > +      <description summary="tool types"/>
> > +      <entry name="pen" value="0x140" summary="pen tool" />
> > +      <entry name="eraser" value="0x141" summary="eraser tool" />
> > +      <entry name="brush" value="0x142" summary="brush tool" />
> > +      <entry name="pencil" value="0x143" summary="pencil tool" />
> > +      <entry name="airbrush" value="0x144" summary="airbrush tool" />
> > +      <entry name="finger" value="0x145" summary="finger tool" />
> > +      <entry name="mouse" value="0x146" summary="mouse tool" />
> > +      <entry name="lens" value="0x147" summary="lens tool" />
> > +    </enum>
> > +
> > +    <event name="proximity_in">
> > +      <description summary="tablet stylus proximity event">
> > +        This event is sent when a tool comes into proximity of the tablet
> > +        above the surface.
> > +
> > +        If the tablet supports tools with hardware serial numbers, the
> > +        tool_serial specifies the serial number. Otherwise, the tool_serial
> > +        number is 0.
> > +      </description>
> > +      <arg name="serial" type="uint"/>
> > +      <arg name="surface" type="object" interface="wl_surface"/>
> > +      <arg name="surface_x" type="fixed" summary="x coordinate in surface-relative coordinates" />
> > +      <arg name="surface_y" type="fixed" summary="y coordinate in surface-relative coordinates" />
> > +      <arg name="axes" type="array"/>
> > +      <arg name="tool" type="uint"/>
> > +      <arg name="tool_serial" type="uint" summary="serial number of the tool, if any"/>
> > +    </event>
> > +
> > +    <event name="proximity_out">
> > +      <description summary="tablet stylus proximity event">
> > +        This event is sent when a tool goes out of proximity or out of the
> > +        surface.
> > +      </description>
> > +      <arg name="serial" type="uint"/>
> > +      <arg name="surface" type="object" interface="wl_surface"/>
> > +    </event>
> > +
> > +    <enum name="button_state">
> > +      <description summary="physical button state">
> > +        Describes the physical state of a button which provoked the button
> > +       event.
> > +      </description>
> > +      <entry name="released" value="0" summary="The button is not pressed"/>
> > +      <entry name="pressed" value="1" summary="The button is pressed"/>
> > +    </enum>
> > +
> > +    <event name="button">
> > +      <description summary="stylus button event">
> > +       Stylus button click and release notifications.
> > +
> > +       The location of the click is given by the last motion or
> > +       proximity event.
> > +        The time argument is a timestamp with millisecond
> > +        granularity, with an undefined base.
> > +      </description>
> > +
> > +      <arg name="serial" type="uint"/>
> > +      <arg name="time" type="uint" summary="timestamp with millisecond granularity"/>
> > +      <arg name="button" type="uint"/>
> > +      <arg name="state" type="uint"/>
> > +    </event>
> > +
> > +    <event name="motion">
> > +      <description summary="tablet stylus motion event"/>
> > +
> > +      <arg name="time" type="uint" summary="timestamp with millisecond granularity"/>
> > +      <arg name="surface_x" type="fixed" summary="x coordinate in surface-relative coordinates" />
> > +      <arg name="surface_y" type="fixed" summary="y coordinate in surface-relative coordinates" />
> > +      <arg name="axes" type="array">
> > +        <description summary="axis state for other axes">
> > +          The state of all absolute axes other than x and y, if any. The
> > +          order of coordinates matches the order of axes as sent by the
> > +          wl_tablet.axis_capability events. x and y coordinates are not
> > +          included, the state starts at the third axis.
> > +        </description>
> > +      </arg>
> > +    </event>
> > +
> > +    <event name="relative_motion">
> > +      <description summary="tablet relative motion event">
> > +        A relative_motion event is sent if a relative axis on a device
> > +        generates an event.
> > +
> > +        This event does not describe motion if the device is in relative
> > +        mode (i.e. like a touchpad), it only notifies of the relative axes
> > +        on the device.
> > +      </description>
> > +
> > +      <arg name="time" type="uint" summary="timestamp with millisecond granularity"/>
> > +      <arg name="surface_x" type="fixed" summary="x coordinate in surface-relative coordinates" />
> > +      <arg name="surface_y" type="fixed" summary="y coordinate in surface-relative coordinates" />
> > +      <arg name="axes" type="array">
> > +        <description summary="axis state for other axes">
> > +          The state of all relative axes other than x and y, if any. The
> > +          order of coordinates matches the order of axes as sent by the
> > +          wl_tablet.axis_capability events. x and y coordinates are not
> > +          included, the state starts at the third axis.
> > +        </description>
> > +      </arg>
> > +    </event>
> > +
> > +    <request name="bind">
> > +      <description summary="bind the tablet to a surface">
> > +        Bind the tablet to the surface. This enables a client to map the
> > +        area of the tablet 1:1 to the surface area. Only one client can bind
> > +        to a tablet at a time.
> 
> Do clients only receive events if they bound the object? If not, once
> a client bound it, do the others still receive events? That is, does
> it work like the evdev-grab thingy?

yes, it is much like the EVIOCGRAB. once bound the client has a straight
connection to the tablet and gets all events (provided the surface is in
focus). So you can say that your GIMP canvas maps directly to your tablet,
regardless of where the canvas is. this is a corner use-case, as I said I'm
not sure this is necessary atm.

otoh it's not like an EVIOCGRAB in that the compositor is still controlling
the device. if the compositor decides the surface doesn't deserve focus at
the moment (e.g. because it's minimised or somesuch), you could still handle
the tablet normally in other clients. but from the client's POV the tablet
is bound, with the upper left of the tablet corresponding to the upper left
of the surface.

> You might also want to add a comment that "binding" is sent as
> response and wl_display.sync may be used as barrier.

can do.

> 
> > +      </description>
> > +      <arg name="surface" type="object" interface="wl_surface" />
> > +    </request>
> > +
> > +    <enum name="bind_status">
> > +      <description summary="notify of bind success"/>
> > +      <entry name="failed" value="0" />
> > +      <entry name="success" value="1" />
> > +    </enum>
> > +
> > +    <event name="binding">
> > +      <description summary="bind notification"/>
> > +      <arg name="status" type="uint"/>
> > +    </event>
> > +
> > +    <request name="unbind">
> > +      <description summary="release the wl_tablet object">
> > +        Unbind the tablet. If the client does not have the tablet bound,
> > +        this request does nothing.
> 
> Nitpick: might want to add: "On object desctruction the tablet is
> implicitly unbound."

yep, will do
> 
> > +      </description>
> > +    </request>
> > +
> > +  </interface>
> > +
> >    <interface name="wl_output" version="2">
> >      <description summary="compositor output region">
> >        An output describes part of the compositor geometry.  The
> 
> Apart from "bind"/"unbind" this really looks to me like a generic
> evdev forwarding. I wonder whether we should try to write a generic
> evdev fallback first and then see whether this is really needed. A
> "bind" might even make sense for the generic interface. Hmmm

it looks like generic forwarding, but over time it'll grow to be more
complicated. wacom requires a bunch of little quirks that you need to take
care of somewhere and I don't even know where exactly to do that.
example: Intuos 3 touch strips come in as ABS_RX/ABS_RY. The wheel comes in
as ABS_WHEEL, the second as ABS_THROTTLE.

I need to go through the wacom driver to check which quirks are X specific
and which aren't. One example that I added locally since XDC is
button-focus (see my reply to Bill in the same thread). that can't be solved
with generic evdev forwarding, at least not without having some extra code
in the compositor.

For example tablet rotation: tablets are rotated to cater
for right/left-handedness and the coordinates need to be adjusted for that
without the client noticing. so simple forwarding doesn't work.

as for the generic evdev fallback: that was my first attempt and it gets
relatively ugly, and krh doesn't want a generic interface anyway. this
approach is a lot nicer and the weston implementation cleaner.

Cheers,
   Peter