[RFC libinput] Add an API for touchpad gesture events
Jonas Ådahl
jadahl at gmail.com
Fri Jan 23 00:51:44 PST 2015
On Fri, Jan 23, 2015 at 09:38:06AM +0100, Hans de Goede wrote:
> Hi,
>
> On 23-01-15 07:18, Jonas Ådahl wrote:
> >On Fri, Jan 23, 2015 at 03:11:31PM +1000, Peter Hutterer wrote:
> >>On Fri, Jan 23, 2015 at 10:11:09AM +0800, Jonas Ådahl wrote:
> >>>On Thu, Jan 22, 2015 at 04:52:50PM +0100, Hans de Goede wrote:
> >>>>For touchscreens we always send raw touch events to the compositor, and the
> >>>>compositor or application toolkits do gesture recognition. This makes sense
> >>>>because on a touchscreen which window / widget the touches are over is
> >>>>important context to know to interpret gestures.
> >>>>
> >>>>On touchpads however we never send raw events since a touchpad is an absolute
> >>>>device which primary function is to send pointer motion delta-s, so we always
> >>>>need to do processing (and a lot of it) on the raw events.
> >>>>
> >>>>Moreover there is nothing underneath the finger which influences how to
> >>>>interpret gestures, and there is a lot of touchpad and libinput configuration
> >>>>specific context necessary for gesture recognition. E.g. is this a clickpad,
> >>>>and if so are softbuttons or clickfinger used? What is the size of the
> >>>>softbuttons? Is this a true multi-touch touchpad or a semi multi-touch touchpad
> >>>>which only gives us a bounding box enclosing the fingers? Etc.
> >>>>
> >>>>So for touchpads it is better to do gesture processing in libinput, this commit
> >>>>adds an initial implementation of a Gesture event API which only supports swipe
> >>>>gestures, other gestures will be added later following the same model wrt,
> >>>>having clear start and stop events and the number of fingers involved being
> >>>>fixed once a gesture sequence starts.
> >>>
> >>>Overall, I think this is probably something we want, but I wonder if
> >>>this conflicts with the wl_touchpad low level Wayland protocol that
> >>>was discussed [0] as part of how to deal with touchpad hand writing (name
> >>>signing, Chinese character writing etc). I.e. *if* we'd provide all the
> >>>data to the client anyway, do we still need to put the gesture processing
> >>>in libinput as well? I'm not saying we shouldn't, just thinking out loud
> >>>whether these are orthogonal or not.
> >>
> >>I think the handwriting bit is a special case that we need to treat
> >>separately anyway, possibly with a mode switch in libinput. gestures can be
> >>supported on top of various touchpad features (software buttons, edge
> >>scrolling), the handwriting is a lot harder because it's only one finger
> >>moving.
> >
> >I suppose so. The thought I had was to just send the touchpoints, and
> >let the rest of the touchpad FSMs do their own thing and let the client
> >ignore it, but that might not the best idea as we'd might end up with
> >stuck buttons or something.
>
> Right, I'm in favor of having a raw touchpad mode for things like handwriting,
> but that really should be a mode switch, the compositor tells libinput to
> put the touchpad in raw-mode and from that point on we only generate raw
> events until switched back.
>
> >>>Anyhow, some comments inlined below:
> >>>
> >>>>
> >>>>Signed-off-by: Hans de Goede <hdegoede at redhat.com>
> >>>>---
> >>>> src/libinput.h | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> >>>> 1 file changed, 154 insertions(+), 1 deletion(-)
> >>>>
> >>>>diff --git a/src/libinput.h b/src/libinput.h
> >>>>index 7b7a2db..1507772 100644
> >>>>--- a/src/libinput.h
> >>>>+++ b/src/libinput.h
> >>>>@@ -173,7 +173,11 @@ enum libinput_event_type {
> >>>> * Signals the end of a set of touchpoints at one device sample
> >>>> * time. This event has no coordinate information attached.
> >>>> */
> >>>>- LIBINPUT_EVENT_TOUCH_FRAME
> >>>>+ LIBINPUT_EVENT_TOUCH_FRAME,
> >>>>+
> >>>>+ LIBINPUT_EVENT_GESTURE_SWIPE_START = 600,
> >>>>+ LIBINPUT_EVENT_GESTURE_SWIPE,
> >>>>+ LIBINPUT_EVENT_GESTURE_SWIPE_END,
> >>>
> >>>This gesture is really similar to our already present scroll gesture,
> >>>and maybe for consistency maybe one single event would be enough? I mean
> >>>that SWIPE_START and SWIPE would be a single one, and SWIPE_END would
> >>>be, as with the scroll gesture, a 0-length vector representing the end
> >>>of a swipe.
>
> I think that the way we're doing start/stop with scrolling is not the best,
> and that having explicit start / stop events is much clearer. E.g. this removes
> the need for the API user to keep some state to determine whether an event
> is a start or an update. I've also been looking at existing gesture API-s
> and this is how Mac OS X does it.
>
> >>>
> >>>A question that comes to mind is what level the swiping should be at.
> >>>For example, these swipe gestures, are they supposed to be simply just
> >>>"swiping" as in N fingers moving together in a single direction and then
> >>>releasing, or more like 2 finger scroll, where the complete motion is
> >>>more unspecified?
>
> These are intended to be really swiping so fingers moving together, the
> initial implementation will likely be more like our current scroll implementation,
> but I do plan to add zoom gesture support later, at least for 2 fingers,
> and then the relative finger position will be taken into account.
>
> >>>If we'd put the swipe gesture in the same "level" as scrolling, we could
> >>>include the 2-fingers case as a gesture as well so that applications
> >>>could make use of it whether the user has enabled 2-fingers scroll or
> >>>not.
>
> Hmm, I don't like overloading stuff like this. I could envision us sending
> 2fg swipe gestures when 2 finger scrolling is disabled, but 2fg scrolling is
> the default, so this gesture won't get emitted much, and thus likely will not
> be used much. And if it is used it will only work for user if they configure
> their touchpad a certain way. No I do not think this is a good idea.
>
> >>>On the other hand, I assume we cannot go too unspecified either, because
> >>>that'd mean a pinch gesture would just be a 2 finger motion gesture where
> >>>the fingers don't stay together. Then again, we could go the middle way
> >>>of saying that for a N fingers motion gesture one need to keep the
> >>>fingers more or less in the same relative distance to each other.
> >>>
> >>>Anyway, do you have any ideas of what level you want to put it?
>
> I want libinput to take care of differentiating between things like a
> zoom vs a swipe gesture. For zoom gestures I'm, thinking to provide
> both a finger-distance-delta and an angle-delta and then the user can decide
> whether to interpret things as a zoom or a rotate, since in some cases one
> or the other may not make sense so the thresholds will be different.
>
> The plan is:
>
> 1) Add swipe support
> 2) Add 2fg zoom
> 3) Start defining a wayland protocol for this
> 4) profit
>
> >>I think for pinch the key is in the trigger to start it, and that could be a
> >>forced separate trigger: i.e. either you move the fingers towards each other
> >>past a certain threshold _or_ you rotate the fingers around some center
> >>point. Not just two fingers moving independently on the touchpad. Once the
> >>gesture is locked in, you can loosen the restrictions.
> >
> >That makes sense. It is the swipe that triggers the gesture, but after
> >the trigger, it has nothing to do with a "swiping" motion any more,
> >except that there is only one motion delta, not N (for N = number of
> >fingers). For pinch and rotate I suppose we'd need both rotation and
> >distance in both of them in order to rotate and zoom at the same time,
> >then again,
>
> Ack, you're pretty much saying what I just wrote, I should have read
> further on before writing that :)
>
> >why wouldn't we want to go from swipe to rotate if we could?
>
> No, this is going to be tricky enough as is, once we've decided for a 2fg
> gesture if it is a swipe(scroll) or a zoom/rotate then we stay in that mode.
I guess so.
>
> >>As for 2 finger swipe at least: isn't that covered by scrolling? with the
> >>caller deciding when a scroll event event is a swipe and when it is a scroll
> >>event. It's the odd one out of course, but scrolling is a special case and
> >>will be the odd one out anyway.
> >
> >Scrolling doesn't completely cover it because a user can disable
> >2-finger scroll for the whole session, and the application won't even
> >see the scroll events. If an application wants to handle the 2-finger
> >swipe no matter what configuration the user has chosen, then it'd
> >probably have to come via something like a gesture API.
>
> AFAIK there is no such thing as a 2fg swipe on other platforms, why would
> we make things difficult to try and support it in the first place ?
Well, don't we already have it fully implemented except we call it
scroll and only enable it if it (although by default) configured so? I
agree though that its not very nice to send out duplicate events for the
same gesture, but still, it feels some how wrong to not let applications
depend on the most simple swipe (deliberately ignoring 1 finger swipe
here) gesture. Also in click-to-scroll configurations, don't we disable
2 finger scroll?
>
> >>>> };
> >>>>
> >>>> /**
> >>>>@@ -350,6 +354,19 @@ libinput_event_get_touch_event(struct libinput_event *event);
> >>>> /**
> >>>> * @ingroup event
> >>>> *
> >>>>+ * Return the gesture event that is this input event. If the event type does
> >>>>+ * not match the gesture event types, this function returns NULL.
> >>>>+ *
> >>>>+ * The inverse of this function is libinput_event_gesture_get_base_event().
> >>>>+ *
> >>>>+ * @return A gesture event, or NULL for other events
> >>>>+ */
> >>>>+struct libinput_event_gesture *
> >>>>+libinput_event_get_gesture_event(struct libinput_event *event);
> >>>>+
> >>>>+/**
> >>>>+ * @ingroup event
> >>>>+ *
> >>>> * Return the device event that is this input event. If the event type does
> >>>> * not match the device event types, this function returns NULL.
> >>>> *
> >>>>@@ -884,6 +901,142 @@ struct libinput_event *
> >>>> libinput_event_touch_get_base_event(struct libinput_event_touch *event);
> >>>>
> >>>> /**
> >>>>+ * @defgroup event_gesture Gesture events
> >>>>+ *
> >>>>+ * Gesture events are generated when a gesture is recognized on a touchpad.
> >>>>+ *
> >>>>+ * Gesture sequences always start with a LIBINPUT_EVENT_GESTURE_FOO_START
> >>>>+ * event. All following gesture events will be of the
> >>>>+ * LIBINPUT_EVENT_GESTURE_FOO type until a LIBINPUT_EVENT_GESTURE_FOO_STOP is
> >>>>+ * generated which signals the end of the gesture.
> >>>>+ */
> >>>>+
> >>>>+/**
> >>>>+ * @ingroup event_gesture
> >>>>+ *
> >>>>+ * @return The event time for this event
> >>>>+ */
> >>>>+uint32_t
> >>>>+libinput_event_gesture_get_time(struct libinput_event_gesture *event);
> >>>>+
> >>>>+/**
> >>>>+ * @ingroup event_gesture
> >>>>+ *
> >>>>+ * @return The generic libinput_event of this event
> >>>>+ */
> >>>>+struct libinput_event *
> >>>>+libinput_event_gesture_get_base_event(struct libinput_event_gesture *event);
> >>>>+
> >>>>+/**
> >>>>+ * @ingroup event_gesture
> >>>>+ *
> >>>>+ * Return the number of fingers used for a gesture. This can be used e.g.
> >>>>+ * to differentiate between 3 or 4 finger swipes.
> >>>
> >>>See above about number of fingers question.
> >>>
> >>>>+ *
> >>>>+ * This function can be called on all gesture events including
> >>>>+ * LIBINPUT_EVENT_GESTURE_FOO_START and the returned finger count value will
> >>>>+ * not change during a sequence. IOW libinput_event_gesture_get_finger_count
> >>>>+ * will only return a different value then a previous call after a
> >>>>+ * LIBINPUT_EVENT_GESTURE_FOO_STOP has been received.
> >>>>+ *
> >>>>+ * @return the number of fingers used for a gesture
> >>>>+ */
> >>>>+enum libinput_key_state
> >>>>+libinput_event_gesture_get_finger_count(struct libinput_event_gesture *event);
> >>>>+
> >>>>+/**
> >>>>+ * @ingroup event_gesture
> >>>>+ *
> >>>>+ * Return the delta between the last event and the current event. For gesture
> >>>>+ * events that are not of type @ref LIBINPUT_EVENT_GESTURE_SWIPE, this
> >>>>+ * function returns 0.
> >>>>+ *
> >>>>+ * If a device employs pointer acceleration, the delta returned by this
> >>>>+ * function is the accelerated delta.
> >>>
> >>>Would we really want this?
>
> Depends on what the swipe is intended to do, for some cases it may make sense
> to have slow & precise vs fast & covering large distances, iow use acceleration
> you're right that we should probably provide unaccelerated values too.
>
> >>>For the 2 fingers scroll, we do not use the
> >>> accelerated motion.
>
> Actually scrolling is exactly a case where we want to support both
> slow & precise and fast & covering large distances, so in
> src/evdev-mt-touchpad.c: tp_post_twofinger_scroll() we've :
>
> tp_filter_motion(tp, &dx, &dy, NULL, NULL, time);
>
> evdev_post_scroll(tp->device,
> time,
> LIBINPUT_POINTER_AXIS_SOURCE_FINGER,
> dx, dy);
>
> And tp_filter_motion() applies acceleration.
Ah, right, my bad.
>
> Regards,
>
> Hans
Jonas
More information about the wayland-devel
mailing list