[RFC libinput] Add an API for touchpad gesture events

Jonas Ådahl jadahl at gmail.com
Thu Jan 22 22:18:55 PST 2015 

On Fri, Jan 23, 2015 at 03:11:31PM +1000, Peter Hutterer wrote:
> On Fri, Jan 23, 2015 at 10:11:09AM +0800, Jonas Ådahl wrote:
> > On Thu, Jan 22, 2015 at 04:52:50PM +0100, Hans de Goede wrote:
> > > For touchscreens we always send raw touch events to the compositor, and the
> > > compositor or application toolkits do gesture recognition. This makes sense
> > > because on a touchscreen which window / widget the touches are over is
> > > important context to know to interpret gestures.
> > > 
> > > On touchpads however we never send raw events since a touchpad is an absolute
> > > device which primary function is to send pointer motion delta-s, so we always
> > > need to do processing (and a lot of it) on the raw events.
> > > 
> > > Moreover there is nothing underneath the finger which influences how to
> > > interpret gestures, and there is a lot of touchpad and libinput configuration
> > > specific context necessary for gesture recognition. E.g. is this a clickpad,
> > > and if so are softbuttons or clickfinger used? What is the size of the
> > > softbuttons? Is this a true multi-touch touchpad or a semi multi-touch touchpad
> > > which only gives us a bounding box enclosing the fingers? Etc.
> > > 
> > > So for touchpads it is better to do gesture processing in libinput, this commit
> > > adds an initial implementation of a Gesture event API which only supports swipe
> > > gestures, other gestures will be added later following the same model wrt,
> > > having clear start and stop events and the number of fingers involved being
> > > fixed once a gesture sequence starts.
> > 
> > Overall, I think this is probably something we want, but I wonder if
> > this conflicts with the wl_touchpad low level Wayland protocol that
> > was discussed [0] as part of how to deal with touchpad hand writing (name
> > signing, Chinese character writing etc). I.e. *if* we'd provide all the
> > data to the client anyway, do we still need to put the gesture processing
> > in libinput as well? I'm not saying we shouldn't, just thinking out loud
> > whether these are orthogonal or not.
> 
> I think the handwriting bit is a special case that we need to treat
> separately anyway, possibly with a mode switch in libinput. gestures can be
> supported on top of various touchpad features (software buttons, edge
> scrolling), the handwriting is a lot harder because it's only one finger
> moving.

I suppose so. The thought I had was to just send the touchpoints, and
let the rest of the touchpad FSMs do their own thing and let the client
ignore it, but that might not the best idea as we'd might end up with
stuck buttons or something.

> 
> > Anyhow, some comments inlined below:
> > 
> > > 
> > > Signed-off-by: Hans de Goede <hdegoede at redhat.com>
> > > ---
> > >  src/libinput.h | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> > >  1 file changed, 154 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/src/libinput.h b/src/libinput.h
> > > index 7b7a2db..1507772 100644
> > > --- a/src/libinput.h
> > > +++ b/src/libinput.h
> > > @@ -173,7 +173,11 @@ enum libinput_event_type {
> > >  	 * Signals the end of a set of touchpoints at one device sample
> > >  	 * time. This event has no coordinate information attached.
> > >  	 */
> > > -	LIBINPUT_EVENT_TOUCH_FRAME
> > > +	LIBINPUT_EVENT_TOUCH_FRAME,
> > > +
> > > +	LIBINPUT_EVENT_GESTURE_SWIPE_START = 600,
> > > +	LIBINPUT_EVENT_GESTURE_SWIPE,
> > > +	LIBINPUT_EVENT_GESTURE_SWIPE_END,
> > 
> > This gesture is really similar to our already present scroll gesture,
> > and maybe for consistency maybe one single event would be enough? I mean
> > that SWIPE_START and SWIPE would be a single one, and SWIPE_END would
> > be, as with the scroll gesture, a 0-length vector representing the end
> > of a swipe.
> > 
> > A question that comes to mind is what level the swiping should be at.
> > For example, these swipe gestures, are they supposed to be simply just
> > "swiping" as in N fingers moving together in a single direction and then
> > releasing, or more like 2 finger scroll, where the complete motion is
> > more unspecified?
> > 
> > If we'd put the swipe gesture in the same "level" as scrolling, we could
> > include the 2-fingers case as a gesture as well so that applications
> > could make use of it whether the user has enabled 2-fingers scroll or
> > not.
> > 
> > On the other hand, I assume we cannot go too unspecified either, because
> > that'd mean a pinch gesture would just be a 2 finger motion gesture where
> > the fingers don't stay together. Then again, we could go the middle way
> > of saying that for a N fingers motion gesture one need to keep the
> > fingers more or less in the same relative distance to each other.
> > 
> > Anyway, do you have any ideas of what level you want to put it?
> 
> I think for pinch the key is in the trigger to start it, and that could be a
> forced separate trigger: i.e. either you move the fingers towards each other
> past a certain threshold _or_ you rotate the fingers around some center
> point. Not just two fingers moving independently on the touchpad. Once the
> gesture is locked in, you can loosen the restrictions.

That makes sense. It is the swipe that triggers the gesture, but after
the trigger, it has nothing to do with a "swiping" motion any more,
except that there is only one motion delta, not N (for N = number of
fingers). For pinch and rotate I suppose we'd need both rotation and
distance in both of them in order to rotate and zoom at the same time,
then again, why wouldn't we want to go from swipe to rotate if we could?
Hmm.

> 
> As for 2 finger swipe at least: isn't that covered by scrolling? with the
> caller deciding when a scroll event event is a swipe and when it is a scroll
> event. It's the odd one out of course, but scrolling is a special case and
> will be the odd one out anyway.

Scrolling doesn't completely cover it because a user can disable
2-finger scroll for the whole session, and the application won't even
see the scroll events. If an application wants to handle the 2-finger
swipe no matter what configuration the user has chosen, then it'd
probably have to come via something like a gesture API.

Jonas

> 
> Cheers,
>    Peter
> > 
> > >  };
> > >  
> > >  /**
> > > @@ -350,6 +354,19 @@ libinput_event_get_touch_event(struct libinput_event *event);
> > >  /**
> > >   * @ingroup event
> > >   *
> > > + * Return the gesture event that is this input event. If the event type does
> > > + * not match the gesture event types, this function returns NULL.
> > > + *
> > > + * The inverse of this function is libinput_event_gesture_get_base_event().
> > > + *
> > > + * @return A gesture event, or NULL for other events
> > > + */
> > > +struct libinput_event_gesture *
> > > +libinput_event_get_gesture_event(struct libinput_event *event);
> > > +
> > > +/**
> > > + * @ingroup event
> > > + *
> > >   * Return the device event that is this input event. If the event type does
> > >   * not match the device event types, this function returns NULL.
> > >   *
> > > @@ -884,6 +901,142 @@ struct libinput_event *
> > >  libinput_event_touch_get_base_event(struct libinput_event_touch *event);
> > >  
> > >  /**
> > > + * @defgroup event_gesture Gesture events
> > > + *
> > > + * Gesture events are generated when a gesture is recognized on a touchpad.
> > > + *
> > > + * Gesture sequences always start with a LIBINPUT_EVENT_GESTURE_FOO_START
> > > + * event. All following gesture events will be of the
> > > + * LIBINPUT_EVENT_GESTURE_FOO type until a LIBINPUT_EVENT_GESTURE_FOO_STOP is
> > > + * generated which signals the end of the gesture.
> > > + */
> > > +
> > > +/**
> > > + * @ingroup event_gesture
> > > + *
> > > + * @return The event time for this event
> > > + */
> > > +uint32_t
> > > +libinput_event_gesture_get_time(struct libinput_event_gesture *event);
> > > +
> > > +/**
> > > + * @ingroup event_gesture
> > > + *
> > > + * @return The generic libinput_event of this event
> > > + */
> > > +struct libinput_event *
> > > +libinput_event_gesture_get_base_event(struct libinput_event_gesture *event);
> > > +
> > > +/**
> > > + * @ingroup event_gesture
> > > + *
> > > + * Return the number of fingers used for a gesture. This can be used e.g.
> > > + * to differentiate between 3 or 4 finger swipes.
> > 
> > See above about number of fingers question.
> > 
> > > + *
> > > + * This function can be called on all gesture events including
> > > + * LIBINPUT_EVENT_GESTURE_FOO_START and the returned finger count value will
> > > + * not change during a sequence. IOW libinput_event_gesture_get_finger_count
> > > + * will only return a different value then a previous call after a
> > > + * LIBINPUT_EVENT_GESTURE_FOO_STOP has been received.
> > > + *
> > > + * @return the number of fingers used for a gesture
> > > + */
> > > +enum libinput_key_state
> > > +libinput_event_gesture_get_finger_count(struct libinput_event_gesture *event);
> > > +
> > > +/**
> > > + * @ingroup event_gesture
> > > + *
> > > + * Return the delta between the last event and the current event. For gesture
> > > + * events that are not of type @ref LIBINPUT_EVENT_GESTURE_SWIPE, this
> > > + * function returns 0.
> > > + *
> > > + * If a device employs pointer acceleration, the delta returned by this
> > > + * function is the accelerated delta.
> > 
> > Would we really want this? For the 2 fingers scroll, we do not use the
> > accelerated motion.
> > 
> > 
> > Jonas
> > 
> > 
> > [0] https://bugs.freedesktop.org/show_bug.cgi?id=85879
> > 
> > > + *
> > > + * Relative motion deltas are normalized to represent those of a device with
> > > + * 1000dpi resolution. See @ref motion_normalization for more details.
> > > + *
> > > + * @note It is an application bug to call this function for events other than
> > > + * @ref LIBINPUT_EVENT_GESTURE_SWIPE.
> > > + *
> > > + * @return the relative x movement since the last event
> > > + */
> > > +double
> > > +libinput_event_gesture_get_dx(struct libinput_event_gesture *event);
> > > +
> > > +/**
> > > + * @ingroup event_gesture
> > > + *
> > > + * Return the delta between the last event and the current event. For gesture
> > > + * events that are not of type @ref LIBINPUT_EVENT_GESTURE_SWIPE, this
> > > + * function returns 0.
> > > + *
> > > + * If a device employs pointer acceleration, the delta returned by this
> > > + * function is the accelerated delta.
> > > + *
> > > + * Relative motion deltas are normalized to represent those of a device with
> > > + * 1000dpi resolution. See @ref motion_normalization for more details.
> > > + *
> > > + * @note It is an application bug to call this function for events other than
> > > + * @ref LIBINPUT_EVENT_GESTURE_SWIPE.
> > > + *
> > > + * @return the relative y movement since the last event
> > > + */
> > > +double
> > > +libinput_event_gesture_get_dy(struct libinput_event_gesture *event);
> > > +
> > > +/**
> > > + * @ingroup event_gesture
> > > + *
> > > + * Return the absolute x coordinate of the current center position of a
> > > + * swipe gesture. This can be used e.g. to determine if a swipe is starting
> > > + * close to a touchpad edge, or to synchronize an animation with how much
> > > + * percent of the width of a touchpad a swipe gesture has travelled.
> > > + *
> > > + * The returned value is in the range of 0.0 to 1.0 with 0.0 indicating that
> > > + * the center is on the right edge of the touchpad, and 1.0 indicating that it
> > > + * is on the left edge.
> > > + *
> > > + * For gesture events that are not of type
> > > + * @ref LIBINPUT_EVENT_GESTURE_SWIPE_START, @ref LIBINPUT_EVENT_GESTURE_SWIPE
> > > + * or @ref LIBINPUT_EVENT_GESTURE_SWIPE_STOP this function returns 0.
> > > + *
> > > + * @note It is an application bug to call this function for events other than
> > > + * @ref LIBINPUT_EVENT_GESTURE_SWIPE_START, @ref LIBINPUT_EVENT_GESTURE_SWIPE
> > > + * or @ref LIBINPUT_EVENT_GESTURE_SWIPE_STOP.
> > > + *
> > > + * @return the current absolute x coordinate
> > > + */
> > > +double
> > > +libinput_event_gesture_get_absolute_x(struct libinput_event_gesture *event);
> > > +
> > > +/**
> > > + * @ingroup event_gesture
> > > + *
> > > + * Return the absolute y coordinate of the current center position of a
> > > + * swipe gesture. This can be used e.g. to determine if a swipe is starting
> > > + * close to a touchpad edge, or to synchronize an animation with how much
> > > + * percent of the height of a touchpad a swipe gesture has travelled.
> > > + *
> > > + * The returned value is in the range of 0.0 to 1.0 with 0.0 indicating that
> > > + * the center is on the top edge of the touchpad, and 1.0 indicating that it
> > > + * is on the bottom edge.
> > > + *
> > > + * For gesture events that are not of type
> > > + * @ref LIBINPUT_EVENT_GESTURE_SWIPE_START, @ref LIBINPUT_EVENT_GESTURE_SWIPE
> > > + * or @ref LIBINPUT_EVENT_GESTURE_SWIPE_STOP this function returns 0.
> > > + *
> > > + * @note It is an application bug to call this function for events other than
> > > + * @ref LIBINPUT_EVENT_GESTURE_SWIPE_START, @ref LIBINPUT_EVENT_GESTURE_SWIPE
> > > + * or @ref LIBINPUT_EVENT_GESTURE_SWIPE_STOP.
> > > + *
> > > + * @return the current absolute y coordinate
> > > + */
> > > +double
> > > +libinput_event_gesture_get_absolute_y(struct libinput_event_gesture *event);
> > > +
> > > +/**
> > >   * @defgroup base Initialization and manipulation of libinput contexts
> > >   */
> > >  
> > > -- 
> > > 2.1.0
> > > 
> > > _______________________________________________
> > > wayland-devel mailing list
> > > wayland-devel at lists.freedesktop.org
> > > http://lists.freedesktop.org/mailman/listinfo/wayland-devel
> > _______________________________________________
> > wayland-devel mailing list
> > wayland-devel at lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/wayland-devel
> >

More information about the wayland-devel mailing list