[RFC DRAFT] graphics tablet protocol extension

[David Herrmann]

Hi Peter

On Wed, Oct 2, 2013 at 11:13 PM, Peter Hutterer
<peter.hutterer at who-t.net> wrote:
> On Wed, Oct 02, 2013 at 05:44:29PM +0200, David Herrmann wrote:
>> Hi Peter
>>
>> On Fri, Sep 20, 2013 at 12:35 PM, Peter Hutterer
>> > diff --git a/protocol/wayland.xml b/protocol/wayland.xml
>> > index aeb0412..8d10746 100644
>> > --- a/protocol/wayland.xml
>> > +++ b/protocol/wayland.xml
>> > @@ -1235,7 +1235,7 @@
>> >      </request>
>> >     </interface>
>> >
>> > -  <interface name="wl_seat" version="3">
>> > +  <interface name="wl_seat" version="4">
>> >      <description summary="group of input devices">
>> >        A seat is a group of keyboards, pointer and touch devices. This
>> >        object is published as a global during start up, or when such a
>> > @@ -1251,6 +1251,7 @@
>> >        <entry name="pointer" value="1" summary="The seat has pointer devices"/>
>> >        <entry name="keyboard" value="2" summary="The seat has one or more keyboards"/>
>> >        <entry name="touch" value="4" summary="The seat has touch devices"/>
>> > +      <entry name="tablets" value="8" summary="The seat has one or more graphics tablet devices" since="4"/>
>>
>> What's actually the reason to allow multiple graphics-tablets per
>> seat? I thought wl_seat objects respresent a single user interacting
>> with your desktop. So if you have two tablets, why not force them to
>> be in two different wl_seat objects? Is there ever a reason to have
>> multiple tablets in a single seat? What would the use-case be?
>
> my laptop has a built-in wacom tablet (lenovo x220t), but I also have an
> Intuos4 plugged in. depending on the use-case I use either, but they should
> be in the same seat. two wl_seat objects means two different keyboard foci
> (only one of which actually works if you have only one keyboard), etc.
> I also know of professional setups that require two tablets or more.
>
> ideally there'd be just one generic wl_tablet per seat, same as wl_pointer
> but the tablets differ too much to make that useable. Hence the multiple
> wl_tablet objects.

Yepp, merging them might be not as easy as with keyboards/mice. Point taken.

>> We could even go further and put tablets into their own seats. Always.
>> A pointer and keyboard may be used by a single user at the same time.
>> But for a graphics-tablet that doesn't sound like a legitimate
>> use-case, does it? But maybe I just have a different view of wl_seat..
>> don't know. I somehow have the feeling we never really agreed on how
>> to define wl_seat objects. Currently they are just user-defined groups
>> of devices with some specific policies ("only one keyboard-object per
>> seat").
>
> my understanding of wl_seat is "one group of input devices". that would
> usually map to one user. it's already not ideal for the bimanual input but
> then again that's even more of a corner case than multiple seats.
>
> once you start moving devices out into other seats, you're in trouble. a
> seat defines keyboard events, so anything that has an interaction with the
> keyboard can't easily be unpaired. a tablet is a fancy pointer device,
> users expect to click somewhere and have it work as a mouse, i.e.
> to move the keyboard focus to wherever you just clicked.
> moving it into a different seat means you'd have to switch to a mouse to
> change the keyboard focus. or, if you have two wl_seats you now need to pair
> them somehow to have the right focus behaviour. and that is going to be
> horrible.

Ok, so wl_seat is just about input focus. I had a much stricter view
of it but obviously it makes it hard to control focus of seats without
mice/keyboards. So yeah, agreed.

>> >      </enum>
>> >
>> >      <event name="capabilities">
>> > @@ -1306,6 +1307,19 @@
>> >        <arg name="name" type="string"/>
>> >      </event>
>> >
>> > +    <!-- Version 4 additions -->
>> > +    <request name="get_tablet_manager" since="4">
>> > +      <description summary="return tablet manager object">
>> > +        The ID provided will be initialized to the wl_tablet_manager
>> > +        interface for this seat. This can then be used to retrieve the
>> > +        objects representing the actual tablet devices.
>> > +
>> > +        This request only takes effect if the seat has the tablets
>> > +        capability.
>> > +      </description>
>> > +      <arg name="id" type="new_id" interface="wl_tablet_manager"/>
>> > +    </request>
>> > +
>> >    </interface>
>> >
>> >    <interface name="wl_pointer" version="3">
>> > @@ -1617,6 +1631,223 @@
>> >      </event>
>> >    </interface>
>> >
>> > +  <interface name="wl_tablet_manager" version="1">
>> > +    <description summary="controller object for graphic tablet devices">
>> > +      A tablet manager object provides requests to access the graphics
>> > +      tablets available on this system.
>> > +    </description>
>> > +
>> > +    <enum name="tablet_type">
>> > +      <description summary="tablet type">
>> > +        Describes the type of tablet.
>> > +      </description>
>> > +      <entry name="external" value="0" summary="The tablet is an external tablet, such as an Intuos" />
>> > +      <entry name="internal" value="1" summary="The tablet is an built-in tablet, usually in a laptop" />
>> > +      <entry name="display" value ="2" summary="The tablet is a display tablet, such as a Cintiq" />
>> > +    </enum>
>> > +
>> > +    <event name="device_added">
>> > +      <description summary="new device notification"/>
>> > +      <arg name="id" type="new_id" interface="wl_tablet" summary="the newly added graphics tablet " />
>> > +      <arg name="name" type="string" summary="the device name"/>
>> > +      <arg name="vid" type="uint" summary="vendor id"/>
>> > +      <arg name="pid" type="uint" summary="product id"/>
>> > +      <arg name="type" type="uint" />
>>
>> So tablets are server-side created objects? I think we tried to avoid
>> that. Only wl_display uses this now, afaik. Even for globals we added
>> the wl_registry interface which just announces unique names which the
>> client then binds to via wl_registry.bind. So why not turn
>> wl_tablet_manager into wl_tablet_registry and provide the same
>> functions as the global registry?
>
> two things:
> having thought about this (and talked to krh and daniels at XDC) the tablet
> manager shouldn't be necessary anymore. It used to have the device_removed
> event, but now it's just an intermediate layer that can be removed by moving
> the device_added event up into the seat and sending it conditional on either
> get_tablets or on binding to the seat with the correct version.

That sounds much better. I like the approach with advertising tablets
on the seat via "tabled_added" events. I don't even think it needs to
be conditional. Older clients just ignore it which seems fine.

>
> as for moving tablets over to a global: my first attempt was to have
> wl_tablet_manager as a global. the server-side objects stay in that case, if
> you bind to the manager it sends you events about the tablets. so for three
> tablets and the client wanting tablet 2 it could look like that:
>
>   wl_tablet_manager::bind
>        event:device added(1)
>        event:device added(2)
>        event:device added(3)
>        request:get_device(some client id, 2)
>
> at this point you still need some sort of ID to identify the tablet when the
> client request comes, so you might as well use server-allocated IDs and let
> the client discard what it doesn't want:
>
>   wl_tablet_manager::bind
>        event:device added(server id 1)
>        event:device added(server id 2)
>        event:device added(server id 3)
>        request:release (server id 1)
>        request:release (server id 3)
>
> so I don't know how to get rid of server-allocated IDs.

Ouh, you always need server-allocated IDs, but not server-allocated
names. What wl_registry does is to provide its own ID namespace for
globals but they are unrelated to Wayland-protocol IDs and just
identify the global. So for tablets you could even reuse the
underlying input-device ID (or event evdev ID) if you wanted. A simple
counter would be enough, though. A client-side "get_tablet(tablet-id,
client-id)" call with the given tablet ID would then allow the client
to create the object itself.

The advantage of your first example is that the client controls the
namespace (client side namespace is _much_ bigger than server-side).
Especially for hotpluggable devices we don't want to risk cluttering
the server-side ID namespace as we have no control over how many of
these may exist. (compared to wl_seat objects which are rather static
and thus could easily be server-side allocated).

wl_global used to be a server-allocated API pre 1.0 then krh changed
it to wl_registry. I don't know his exact reasons but I think it's to
try to avoid server-side allocated global IDs at all. But obviously we
should ask him. This is just my understanding of it.

> as said above, the tablet_manager doesn't really provide that much
> functionality at this point, so another alternative is to have wl_tablet
> directly as global, but I don't know how to hook that up with seats
> then. and you'd still have server-allocated IDs.

Yepp, if it's a global you get problems binding it to a seat. Don't
think this is a good idea.

>> > +    </event>
>> > +
>> > +  </interface>
>> > +
>> > +  <interface name="wl_tablet" version="1">
>> > +    <description summary="a graphics tablet device">
>> > +      This interface describes a graphics tablet, such as Wacom's Intuos or
>> > +      Cintiq ranges.
>> > +
>> > +      Note, this interface does not handle touch input on touch-capable
>> > +      tablets. Depending on the tablet, touch input is sent through the
>> > +      wl_pointer (if used as a touchpad) or wl_touch interface (if used as a
>> > +      touch screen).
>> > +    </description>
>> > +    <enum name="axis_type">
>> > +      <description summary="axis type"/>
>> > +      <entry name="relative" value="2" summary="relative axis"/>
>> > +      <entry name="absolute" value="3" summary="relative axis"/>
>> > +    </enum>
>> > +
>> > +    <request name="describe">
>> > +      <description summary="request axis information">
>> > +        Request axis information from a device. Whe a client sends a
>> > +        wl_tablet.describe request, the compositor will emit a
>> > +        wl_tablet.axis_capability event for each axis on the device.
>> > +        To mark the end of the burst of events, the client can use the
>> > +        wl_display.sync request immediately after calling wl_tablet.describe.
>> > +      </description>
>>
>> Why not always send the axis information? I mean once the client
>> requests a tablet_manager doesn't it always want to retrieve a
>> description? A wl_display.sync directly after receiving device_added
>> would act as the barrier. The only reason not to send this information
>> would be to safe traffic if multiple tablets are connected and the
>> client wants only one of them. Does this really justify the additional
>> method? I have no idea how big these data-sets really are..
>
> tablets atm have a couple of buttons and a couple of axes, but nothing
> major (iirc it's 10 or less, depending on the tablet). the reason for the
> separate request is that the device_added event has some device information
> (notably PID/VID and type), so that clients can simply ignore the ones they
> don't want without having to deal with axis description events as well.

If you use server-allocated objects, then you're right. If you use
client-allocated objects you only send the events once the client
really _creates_ the object (instead of on event generation). So yeah,
depends on which model we choose.

>
>> > +    </request>
>> > +
>> > +    <request name="release" type="destructor">
>> > +      <description summary="release the wl_tablet object"/>
>> > +    </request>
>> > +
>> > +
>>
>> Nitpick: two blank lines
>>
>> > +    <event name="removed">
>> > +      <description summary="the device has been removed"/>
>> > +    </event>
>> > +
>> > +    <event name="axis_capability">
>> > +      <description summary="axis description">
>> > +        For relative devices, min and max are 0. A resolution, fuzz, or flat of 0
>> > +        denotes an unknown resolution, fuzz, or flat, respectively.
>> > +
>> > +        Note that while the min/max values are announced by the device, the value is
>> > +        not clamped to the actual range by the compositor. A device may send values
>> > +        outside this range.
>> > +      </description>
>> > +      <arg name="type" type="uint" summary="axis type (relative, absolute)" />
>> > +      <arg name="axis" type="uint" summary="axis identifier" />
>> > +      <arg name="min" type="int" summary="minimum axis value, inclusive"/>
>> > +      <arg name="max" type="int" summary="maximum axis value, inclusive"/>
>> > +      <arg name="resolution" type="uint" summary="resolution in units/mm"/>
>> > +      <arg name="fuzz" type="uint"/>
>> > +      <arg name="flat" type="uint"/>
>> > +    </event>
>> > +
>> > +    <event name="button_capability">
>> > +      <description summary="button description">
>> > +      </description>
>> > +      <arg name="buttons" type="array" />
>> > +    </event>
>> > +
>> > +    <enum name="tool_type">
>> > +      <description summary="tool types"/>
>> > +      <entry name="pen" value="0x140" summary="pen tool" />
>> > +      <entry name="eraser" value="0x141" summary="eraser tool" />
>> > +      <entry name="brush" value="0x142" summary="brush tool" />
>> > +      <entry name="pencil" value="0x143" summary="pencil tool" />
>> > +      <entry name="airbrush" value="0x144" summary="airbrush tool" />
>> > +      <entry name="finger" value="0x145" summary="finger tool" />
>> > +      <entry name="mouse" value="0x146" summary="mouse tool" />
>> > +      <entry name="lens" value="0x147" summary="lens tool" />
>> > +    </enum>
>> > +
>> > +    <event name="proximity_in">
>> > +      <description summary="tablet stylus proximity event">
>> > +        This event is sent when a tool comes into proximity of the tablet
>> > +        above the surface.
>> > +
>> > +        If the tablet supports tools with hardware serial numbers, the
>> > +        tool_serial specifies the serial number. Otherwise, the tool_serial
>> > +        number is 0.
>> > +      </description>
>> > +      <arg name="serial" type="uint"/>
>> > +      <arg name="surface" type="object" interface="wl_surface"/>
>> > +      <arg name="surface_x" type="fixed" summary="x coordinate in surface-relative coordinates" />
>> > +      <arg name="surface_y" type="fixed" summary="y coordinate in surface-relative coordinates" />
>> > +      <arg name="axes" type="array"/>
>> > +      <arg name="tool" type="uint"/>
>> > +      <arg name="tool_serial" type="uint" summary="serial number of the tool, if any"/>
>> > +    </event>
>> > +
>> > +    <event name="proximity_out">
>> > +      <description summary="tablet stylus proximity event">
>> > +        This event is sent when a tool goes out of proximity or out of the
>> > +        surface.
>> > +      </description>
>> > +      <arg name="serial" type="uint"/>
>> > +      <arg name="surface" type="object" interface="wl_surface"/>
>> > +    </event>
>> > +
>> > +    <enum name="button_state">
>> > +      <description summary="physical button state">
>> > +        Describes the physical state of a button which provoked the button
>> > +       event.
>> > +      </description>
>> > +      <entry name="released" value="0" summary="The button is not pressed"/>
>> > +      <entry name="pressed" value="1" summary="The button is pressed"/>
>> > +    </enum>
>> > +
>> > +    <event name="button">
>> > +      <description summary="stylus button event">
>> > +       Stylus button click and release notifications.
>> > +
>> > +       The location of the click is given by the last motion or
>> > +       proximity event.
>> > +        The time argument is a timestamp with millisecond
>> > +        granularity, with an undefined base.
>> > +      </description>
>> > +
>> > +      <arg name="serial" type="uint"/>
>> > +      <arg name="time" type="uint" summary="timestamp with millisecond granularity"/>
>> > +      <arg name="button" type="uint"/>
>> > +      <arg name="state" type="uint"/>
>> > +    </event>
>> > +
>> > +    <event name="motion">
>> > +      <description summary="tablet stylus motion event"/>
>> > +
>> > +      <arg name="time" type="uint" summary="timestamp with millisecond granularity"/>
>> > +      <arg name="surface_x" type="fixed" summary="x coordinate in surface-relative coordinates" />
>> > +      <arg name="surface_y" type="fixed" summary="y coordinate in surface-relative coordinates" />
>> > +      <arg name="axes" type="array">
>> > +        <description summary="axis state for other axes">
>> > +          The state of all absolute axes other than x and y, if any. The
>> > +          order of coordinates matches the order of axes as sent by the
>> > +          wl_tablet.axis_capability events. x and y coordinates are not
>> > +          included, the state starts at the third axis.
>> > +        </description>
>> > +      </arg>
>> > +    </event>
>> > +
>> > +    <event name="relative_motion">
>> > +      <description summary="tablet relative motion event">
>> > +        A relative_motion event is sent if a relative axis on a device
>> > +        generates an event.
>> > +
>> > +        This event does not describe motion if the device is in relative
>> > +        mode (i.e. like a touchpad), it only notifies of the relative axes
>> > +        on the device.
>> > +      </description>
>> > +
>> > +      <arg name="time" type="uint" summary="timestamp with millisecond granularity"/>
>> > +      <arg name="surface_x" type="fixed" summary="x coordinate in surface-relative coordinates" />
>> > +      <arg name="surface_y" type="fixed" summary="y coordinate in surface-relative coordinates" />
>> > +      <arg name="axes" type="array">
>> > +        <description summary="axis state for other axes">
>> > +          The state of all relative axes other than x and y, if any. The
>> > +          order of coordinates matches the order of axes as sent by the
>> > +          wl_tablet.axis_capability events. x and y coordinates are not
>> > +          included, the state starts at the third axis.
>> > +        </description>
>> > +      </arg>
>> > +    </event>
>> > +
>> > +    <request name="bind">
>> > +      <description summary="bind the tablet to a surface">
>> > +        Bind the tablet to the surface. This enables a client to map the
>> > +        area of the tablet 1:1 to the surface area. Only one client can bind
>> > +        to a tablet at a time.
>>
>> Do clients only receive events if they bound the object? If not, once
>> a client bound it, do the others still receive events? That is, does
>> it work like the evdev-grab thingy?
>
> yes, it is much like the EVIOCGRAB. once bound the client has a straight
> connection to the tablet and gets all events (provided the surface is in
> focus). So you can say that your GIMP canvas maps directly to your tablet,
> regardless of where the canvas is. this is a corner use-case, as I said I'm
> not sure this is necessary atm.
>
> otoh it's not like an EVIOCGRAB in that the compositor is still controlling
> the device. if the compositor decides the surface doesn't deserve focus at
> the moment (e.g. because it's minimised or somesuch), you could still handle
> the tablet normally in other clients. but from the client's POV the tablet
> is bound, with the upper left of the tablet corresponding to the upper left
> of the surface.

But binding doesn't do scaling, right? So the upper-left is bound, but
that doesn't mean the lower-right is bound to the lower-right corner,
does it? Would be weird, I guess.

Regarding Grab: So does this mean the server _can_ actually allow two
clients to bind to a tablet if one is minimized? Or does it just mean
the server is allowed to use the tablet itself if the client is
minimized? Because your comment clearly states only one client can
bind at a time.

Just as a remark, DirectInput on Windows only allows clients with
input-focus (here: of the respective wl_seat) to bind to devices. Once
it looses focus, the binding is lost. This allows clients to share
devices but also to get exclusive access if active/focused.

>> You might also want to add a comment that "binding" is sent as
>> response and wl_display.sync may be used as barrier.
>
> can do.
>
>>
>> > +      </description>
>> > +      <arg name="surface" type="object" interface="wl_surface" />
>> > +    </request>
>> > +
>> > +    <enum name="bind_status">
>> > +      <description summary="notify of bind success"/>
>> > +      <entry name="failed" value="0" />
>> > +      <entry name="success" value="1" />
>> > +    </enum>
>> > +
>> > +    <event name="binding">
>> > +      <description summary="bind notification"/>
>> > +      <arg name="status" type="uint"/>
>> > +    </event>
>> > +
>> > +    <request name="unbind">
>> > +      <description summary="release the wl_tablet object">
>> > +        Unbind the tablet. If the client does not have the tablet bound,
>> > +        this request does nothing.
>>
>> Nitpick: might want to add: "On object desctruction the tablet is
>> implicitly unbound."
>
> yep, will do
>>
>> > +      </description>
>> > +    </request>
>> > +
>> > +  </interface>
>> > +
>> >    <interface name="wl_output" version="2">
>> >      <description summary="compositor output region">
>> >        An output describes part of the compositor geometry.  The
>>
>> Apart from "bind"/"unbind" this really looks to me like a generic
>> evdev forwarding. I wonder whether we should try to write a generic
>> evdev fallback first and then see whether this is really needed. A
>> "bind" might even make sense for the generic interface. Hmmm
>
> it looks like generic forwarding, but over time it'll grow to be more
> complicated. wacom requires a bunch of little quirks that you need to take
> care of somewhere and I don't even know where exactly to do that.
> example: Intuos 3 touch strips come in as ABS_RX/ABS_RY. The wheel comes in
> as ABS_WHEEL, the second as ABS_THROTTLE.
>
> I need to go through the wacom driver to check which quirks are X specific
> and which aren't. One example that I added locally since XDC is
> button-focus (see my reply to Bill in the same thread). that can't be solved
> with generic evdev forwarding, at least not without having some extra code
> in the compositor.
>
> For example tablet rotation: tablets are rotated to cater
> for right/left-handedness and the coordinates need to be adjusted for that
> without the client noticing. so simple forwarding doesn't work.
>
> as for the generic evdev fallback: that was my first attempt and it gets
> relatively ugly, and krh doesn't want a generic interface anyway. this
> approach is a lot nicer and the weston implementation cleaner.

Ok, understood. My tablet knowledge is a little bit rusty..

Thanks
David