X and gestures architecture review

[Peter Hutterer]

should have read this one before writing the reply to the other email :)
anyway:

On Fri, Aug 27, 2010 at 12:15:08PM -0400, Chase Douglas wrote:
> I've been formulating some thoughts about gesture recognition in an X
> environment over the past week. I haven't had time to sit down and write
> them until now due to the Maverick beta freeze deadline yesterday.
> 
> I think there's a fundamental issue that would be worth hammering out
> above all other issues with the proposed X Gesture extension: should
> gesture recognition occur server side, before events are propagated to
> clients?

think about it in a different way: does gesture recognition take precedence
over multi-touch?

To me, the answer is no and that implies that touch points must be handed to
the client before and gesture interpretation can interfer.
 
> I've been doing some thinking based on feedback from Peter Hutterer. He
> is of the mindset that gestures should be recognized on
> the client side only, whereas the X Gesture extension recognizes
> gestures on the server side. I've been trying to think of the reasons
> for us going with server side handling so far, and here's what I've come
> up with:
> 
> 1. Say you have two windows side by side and you do a four finger drag
> with two fingers over each window. With X Gesture, the four fingers
> would be recognized together and sent only to the root window, where
> a window manager might be listening. This is because all fingers of a
> gesture must start within the same window, and the recognition occurs
> without any context of where windows are on screen. One could view this
> as a policy decision we have made.
> 
> 2. If recognition occurs client side, it needs to occur twice so that a
> window manager can attempt recognition and replay events to clients who
> also attempt recognition. Anecdotally, I think our utouch-grail gesture
> engine is fast enough to do recognition twice with little latency, but
> it wouldn't seem to be the most optimal solution.

I wonder if someone did some research on perceived latency between
direct-touch and gestures. My guess is that the tolerance for latency in
gestures is higher than for direct-touch interfaces but it'd be nice to see
some published results on that. If anyone knows of any, please let me know.

> 3. We don't have access to MT events on the client side in Maverick.
> 
> I think I've been conflating the third issue with general gesture
> recognition and propagation logic to the point that when I look back
> it's hard to remember exactly why we need recognition to be server side
> due to point one alone. I know that we have to do gestures through X in
> Maverick due to a lack of MT events, but maybe that's been coloring my
> view on how it should be done in the future when MT events are
> available?
> 
> It's true that the logic behind point one may be perfectly fine, but
> having recognition through X inserts a lot of extra code into the
> server. If we are content with touches being split up into window
> regions before recognition occurs, then we may be able to get around the
> need for the X Gesture extension completely. The window manager use case
> could be supplied through input grabbing and event replaying.
> 
> However, what's the best way to resolve point 2?
> 
> We have not yet begun development for the next Ubuntu cycle, but we will
> be tackling it shortly. We are open to trying out any approach that
> seems reasonable, whether it's client side or server side recognition.
> At this point I'm on the fence due to the amount of work required to
> implement the X gesture extension vs the potential latencies encountered
> by gesture recognition being performed twice.

I think right now getting MT events to the clients would be the most
productive way to invest your spare development time right now, especially
since any proposed gesture implementation would need to build on top of the
server understanding multitouch anyway.

There's bound to be some issues in the current MT proposals but it's likely
we won't see all ofh them them until we implement it (and the
client-support). Upstream development doesn't always play nice with
distribution releases, so focusing on getting it right upstream will have a
better long-term result than trying to squeeze something into the
development cycle of a specific distribution release.

Cheers,
  Peter