Adding support for a (multi-)touchscreen - where best to add it?

[Peter Hutterer]

On Fri, Mar 13, 2009 at 06:21:04AM -0500, David Hagood wrote:
> On Fri, 2009-03-13 at 11:43 +1000, Peter Hutterer wrote:
> > On Thu, Mar 12, 2009 at 01:40:29PM -0400, Chris Ball wrote:
> 
> > I agree. getting something sensible in the kernel prevents us from replicating
> > the "one X driver per device" hilarity.
> > Once we have that support in the kernel, we can look at how to get it to
> > applications in a sensible manner. Real multi-touch is hard, and there are a
> > few roadblocks that I'm not looking forward to.
> > 
> > Cheers,
> >   Peter
> 
> OK, let me break it down into steps:
> 
> Step 1: add the code to the kernel HID system to
>   1a) identify this device as being a special case within the HID system
>   1b) send the device the needed special packet to tell it to start
> sending data (otherwise it does nothing).
>   1c) Parse the incoming data (it uses a specific format) into HID
> events.
> 
> Alternatively 1a-1c could be done in a userspace library e.g. tslib.
> 
> Step 2: Add support at the X input layer to handle those packets, in one
> of the several candidate drivers (e.g. input-tslib, input-evtouch,
> input-evdev).
>   As I understand it the consensus here is that this should be in evdev.
> NOTE: currently the other touchscreens I have from 3M and Dannotech do
> NOT work under evdev - when evdev is bound to them it doesn't generate
> any events to X. They do work under tslib and evtouch.

that's probably because they advertise a button/axis combination that evdev
does interpret correctly, or because they're posting the axis information
wrongly. I had my hands on an HP Touchsmart for a short while and it was
posting x/y through Z/Rx.
 
> Step 3: Hook into the multipointer support in X itself.
>   As I understand it X should already support multiple pointers now.
> 
> Step 4: Hook into MP support in the toolkit (e.g. Qt) layers.
>   As I understand it several of the toolkits support MP now.
> 
> Is that right?

sort-of. except that there's a huge difference between multi-touch,
multi-point and multi-touch-multi-point.

I'm ignoring the GUI challenges and just focus on the technical issues:

single point: this is what we have now. single-point interaction. One x/y
coordinate, button/key state.

touch: not a single point, but a single area. touching with a thumb is
different than touching with a stylus. touch screens that only to single
point interaction are technically no different to a mouse.

multi-point: gives you multiple (semantical) points of interaction. The
troubles here are that all apps expect single point, so you need to isolate
the points of interaction to some extent. This is what we have in git X now.
it gets tricky when you start dealing with multiple points that belong to a
number of users, not necessarily the same number of points.

multi-touch - single-point: gives you multiple areas but they are semantically
one input point. This is the easy one. you run one app full-screen and get the
data. it belongs to one user, and you interpret it accordingly (e.g. "pinch"
gesture) once we have kernel support for this, you can just forward this
information to an app and be happy. 
or not. because what you really want is multi-touch + multi-point.

multi-touch + multi-point: you have multiple separate input points that all
are multi-touch capable. which allows you to do a pinch gesture with one hand
while you're rotating an object with your other hand in a different client.

This is hard.

>From the list of touch ares that you get, how do you separate input points?
If you're not running fullscreen, how do you do the picking?
Even worse, you may not even know until well into the gesture. Because what
looked like a pinch gesture at first is actually two index fingers of two
users that both tried to move an object towards them. Oops.

Oh, and there's also some fun in providing touch support in standard apps that
think that your fingers are a mouse.

Cheers,
  Peter

PS: google for multi-touch demonstration and try to find one that doesn't just
use one full-screen app. Not a lot of them around.