RFC: Wayland high-resolution wheel scrolling
Peter Hutterer
peter.hutterer at who-t.net
Sat Apr 6 00:22:54 UTC 2019
On 6/4/19 02:11 , Simon Ser wrote:
> On Wednesday, April 3, 2019 5:36 AM, Peter Hutterer <peter.hutterer at who-t.net> wrote:
>> I've been trying to sort out the new hi-res wheel scrolling that was added to
>> linux 5.0 but it's been a bit of a struggle to say the least. For the
>> impatient, skip forward to the protocol diff but the background info is
>> important.
>
> Thanks for taking the time to write this big summary. It's pretty helpful!
>
>> Everything below applies equally to horizontal and vertical scrolling, both
>> directions and wheels and wheel tilts.
>>
>> The APIs that are involved for low-resolution wheel scrolling:
>>
>> kernel:
>> - REL_WHEEL for every wheel click
>>
>> libinput uses pointer axis events for wheels:
>> - pointer axis value: the movement of the wheel in degrees
>> - pointer axis discrete value: the number of wheel clicks
>> - pointer axis source: set to 'wheel'
>>
>> Usally this means you get discrete 1, value 15 for the standard 15-degree mouse
>> wheel, discrete 2 value 30 is two wheel clicks within one hw frame.
>>
>> There is one implicit assumption here: the discrete value must
>> be 1 or more for every event. Otherwise it's impossible to use the value and
>> calculate the angle. More on that later.
>>
>> This API is effectively the same in the wayland protocol except for the unit of
>> the 'value' which is is in screen coordinates, not degrees.
>>
>> wayland protocol:
>> - pointer axis value: the movement in screen coordinates
>> - pointer axis discrete: the number of wheel clicks
>> - pointer axis source: set to 'wheel'
>>
>> The big difference here though: libinput's API is per-device, wayland's API is
>> per wl-seat pointer. There's no way to tell in the protocol that you're
>> scrolling simultaneously with wheels on two different mice.
>>
>> Let's see how the common compositors implement this:
>>
>> weston:
>> - for source wheel, only the discrete value is handled and:
>> - pointer axis discrete is sent as-is from libinput
>> - pointer axis source is sent as-is from libinput
>> - pointer axis value is always (discrete * 10)
>> - otherwise the libinput pointer value (degrees) from a source wheel is ignored
>>
>> The magic multiplier 10 is afaik for historical reasons, weston used to do 10
>> so early libinput took that. When libinput switched to use physical dimensions
>> instead of 10, that value was kept for backwards compatibility with existing
>> wayland clients.
>>
>> mutter:
>> - for source wheel, only the discrete value is handled and:
>> - clutter-internal event for the discrete **direction** from libinput
>> - i.e. discrete > 1 will still only generate one event,some events
>> drop to the floor here for fast wheel movements
>> - clutter-internal event with (value * 10) marked as 'emulated' but
>> that event is ignored later when we're writing to the protocol
>> - the discrete event is written as wl_pointer.axis_discrete and
>> wl_pointer.axis with a value of 10
>>
>> So, the same multiplier of 10 but fast scroll events with more than one click
>> per frame appear to get dropped.
>>
>> kwin (through qtwayland):
>> - doesn't handle discrete at all afaict
>> - pointer axis events are passed through but I got a bit confused over the
>> actual value written on the wire here.
>>
>> wlroots:
>> - pointer axis discrete is sent as-is from libinput
>> - pointer axis source is sent as-is from libinput
>> - pointer axis value is sent as-is from libinput
>>
>> Because the physical angle is passed on from libinput as screen coordinates,
>> this means wlroots has different scroll behaviour to GNOME/sway and it does
>> scroll a larger distance for wheels with higher click angles. This is the
>> opposite of the intended behaviour, manufacturers advertise these wheels
>> as slow-scrolling wheels. Realistically though the physical movements is the
>> same-ish and if a client uses discrete it doesn't matter anyway.
>
> Hmm, that's interesting. We wlroots devs have had assumed you'd want a given
> angle to always scroll the same amount of pixels. But this doesn't matter that
> much because we have a configuration option for the scroll multiplier anyway.
> We still should have good defaults of course.
first, apologies if i got some of the details wrong, I hope the overall
summary is still correct enough :)
the thing with the angle in this case is that you cannot really decide
that angle. The wheel forces you to whatever it's built as and that's it
(more recent high-res wheels excepted). Even the earlier free-floating
wheels still only send events every X degrees. Users (and manufacturers)
thus don't think as angles as much as wheel clicks. A 20 deg wheel feels
slower than the standard 15 deg wheel simply because you have to scroll
more to get one click. And that's what's advertised too.
The angle == pixels mapping makes sense for a continuous axis, I'm not
sure it's as useful for the wheel as we have them. Then again, it's
relatively niche and only a small subset of users notice and/or care
about this particular issue.
>> xf86-input-libinput:
>> - X devices need to set up a scroll distance for scroll motion, the driver
>> uses 120 (since Feb, before it was 15). This means any multiple of 120.0
>> triggers legacy button emulation in the server, otherwise the number has no
>> meaning.
>> - for source wheel it uses the discrete value only, so effectively 120 *
>> discrete.
>>
>> This driver had a division by 0 for discrete values of 0, fixed in the
>> current release. More on this later.
>>
>>
>>
>> Let's look at how wheel events are processed in some of the clients:
>>
>> GTK:
>> - for a discrete event, GDK_SCROLL_UP is passed on flagged as 'emulated'
>> - the axis value is passed on as GDK_SCROLL_SMOOTH
>>
>> Qt:
>> - doesn't handle discrete
>> - multiplies the value by -12. This gives it a 120-based value which
>> matches the Windows API provided the axis value is 10 of course.
>> Qt on wlroots is probably off here.
>>
>> Xwayland:
>> - X devices need to set up a scroll distance for scroll motion, Xwayland
>> currently uses 1.0 for that. This means any multiple of 1.0 triggers legacy
>> button emulation in the server.
>> - wl_pointer.axis_discrete are used where available
>> - wl_pointer.axis uses the value * 0.1. This effectively means Xwayland
>> relies on the deltas to be 10 like in mutter/weston, X clients under
>> wlroots will have off wheel click emulation.
>
> Xwayland uses axis_discrete when available, so wlroots should be fine (we sent
> the patch doing that when we decided not to implement axis like Weston IIRC).
yep, this must've been scrambled notes by me. xwayland uses discrete but
where discrete is missing, the axis value is missing it uses the 0.1. So
if we were to drop axis discrete and rely on axis only, the 10 vs
$click-angle mismatch will cause weird effects.
>
>> With me so far? Hooray, now let's introduce hi-res scrolling.
>>
>> All the examples below are for a quarter scroll wheel movement, i.e. you get
>> 4 hi-res events for every 1 lo-res event. The kernel uses a fraction/multiple
>> of 120 for logical clicks, same as the Windows API. So one wheel click is 120,
>> half a click is 60, etc. I'm calling this 120-based value the v120 because I'm
>> very creative.
>>
>> kernel:
>> - REL_WHEEL for every wheel click
>> - REL_WHEEL_HI_RES for every fraction of a wheel click (as portion of 120),
>> so a quarter-wheel stop gives you a value of 30.
>> - there is no guarantee that REL_WHEEL is sent every full 120 because e.g.
>> Logitech mice may reset halfway through a wheel motion. So the two axes
>> must be considered completely independent.
>>
>> libinput:
>>
>> Basically: the current API is fairly useless in handling fractional scroll
>> because libinput uses physical distances. libinput could send four events:
>> - value 3.75, discrete 0
>> - value 3.75, discrete 0
>> - value 3.75, discrete 0
>> - value 3.75, discrete 1
>>
>> Which is technically correct. But callers have no reliable way how much of a
>> fraction of a wheel the 3.75 represents until the first discrete event comes
>> in. Guessing is unreliable, if you scroll faster you may get 2+ fractional
>> units in one event. And mice may reset halfway through a scroll motion so you
>> don't always get the same number of events per discrete. So libinput needs some
>> extra API that gives us some baseline to compare values against, i.e. same as
>> the kernel's 120-based API does.
>>
>> There are other problems, specifically related to sending discrete 0 events:
>>
>> weston:
>> - Current weston: a discrete value of 0 ends up as value of 0 which weston
>> treats as axis_stop event. So our event sequence is axis_source,
>> axis_stop, pointer_frame. this breaks scrolling.
>>
>> mutter:
>> - Current mutter: a discrete value of 0 ends up in noops.
>> mutter still generates internal clutter events but they are never pushed
>> onto the wire. The normal values are ignored, so we basically
>> get low-res scrolling just as before.
>> Note: this is based on reading the code, not test runs
>>
>> wlroots:
>> - will forward value/discrete as it comes in from libinput
>
> We're in the same boat as Weston here: the special value 0 will be sent as
> axis_stop.
>
>>
>> kwin:
>> - doesn't use discrete so won't be affected by changes
>>
>> xf86-input-libinput:
>> - anything pre 0.28.2 will get a division by 0
>> - current state: discrete values of 0 end up as noops and we we get lo-res
>> scrolling only
>>
>> So basically: if we change libinput to send discrete 0 events, we'll break
>> weston and all but the most recent xf86-input-libinput. Code changes are needed
>> just to cope with the different event sequence, let alone the actual hi-res bits.
>
> Agreed.
>
>> The solution to this is to add a new event, LIBINPUT_EVENT_POINTER_AXIS_WHEEL.
>> That event provides libinput_event_pointer_axis_value_v120() which is basically
>> a mirror of the kernel API. This event obsoletes POINTER_AXIS events for
>> sources WHEEL and WHEEL_TILT, so callers should just ignore those if they
>> support the new event. A branch for this is available here:
>> https://gitlab.freedesktop.org/whot/libinput/commits/wip/hi-res-scrolling
>>
>> The event sequence would now be:
>>
>> - WHEEL: value 3.75, v120 30
>> - WHEEL: value 3.75, v120 30
>> - WHEEL: value 3.75, v120 30
>> - WHEEL: value 3.75, v120 30
>> - AXIS: value 3.75, discrete 1, v120 120
>>
>> Implementation-wise: AXIS and WHEEL are independent, so there's no guarantee
>> that they add up to the same 120 values. This new event is easy to add to
>> compositors.
>
> This makes sense to me. It's unfortunate to introduce a new API, but I guess as
> discussed before there's no way around it.
>
>> Let's look at the Wayland protocol with the current API given hi-res scroll
>> events:
>> - value 3.75, no discrete event
>> - value 3.75, no discrete event
>> - value 3.75, no discrete event
>> - value 3.75, discrete 1
>>
>> This is assuming 15 degrees/4 == 3.75 but if the compositor force wheels
>> to a scale of 10, the value would be 2.5.
>>
>> Qt:
>> - doesn't use discrete so won't be affected by changes
>>
>> GTK:
>> - generates 4 GDK_SCROLL_SMOOTH events,
>> - generates 1 GDK_SCROLL_DOWN event ('emulated')
>> - GTK doesn't really care as such about whether the distance-to-wheel is 10 or
>> something else, it'll just have different scroll distances (e.g. under
>> wlroots).
>>
>> Afaict this works correctly (tested with stock F29 nautilus)
>>
>> Xwayland:
>> - current Xwayland: values are handled as fraction as before, so we get 3
>> smooth-scroll events at 25%. Event 4 has a discrete event which is handled
>> as full event, so we scroll by a logical click. Total distance is 175%.
>
> Hmm. Is a logical click 150%? I think I'm missing something here.
yeah, I wasn't clear enough. A full wheel would cause 3 smooth events
and 1 discrete one, ending up in 25% + 25% + 25% + 100% = 175% total. If
we were to keep the discrete at 0, it'd be 25% for the last event too
and add up to the correct 100%.
> But yeah, if we stop sending axis_discrete and continue sending axis, it'll
> break clients.
Writing the above, now I'm wondering: if we deprecate axis_discrete and
stop sending them, I think clients will keep working (they have to,
axis-discrete were never guaranteed). So we *might* be able to replace
the discrete events with something new like the v120 event and be good.
Need to look into that.
>> This means: Xwayland cannot work at all without any extra wayland protocol
>> unless it assumes factor 10 is the base unit.
>>
>>
>> So we have at least one ubiquitous Wayland client that cannot handle it.
>> And the issues are very similar to the ones libinput's API has, added to that
>> is that you cannot guarantee that two events come from the same device.
>> What we need is a new event, but I'm struggling to add this to the wayland
>> protocol in a good way, mostly for backwards-compatibility reasons.
>>
>>
>> Much of the below would be easier to solve if we can say "if you claim to
>> support wl_pointer version 8, you get different behaviour for axis events". I
>> don't think we can trust clients to handle this correctly, I got burned by
>> those assumptions before (in XI2). So I'm ruling out changing any of the
>> existing protocol behaviour.
>
> Hmm. We have to be careful -- anything we decide will be set in stone.
>
> The protocol would look less ugly if we change wl_pointer's meaning in a new
> version. (A similar change has already been done to the wl_data_offer API)
> Though it's true that until we still have users of the old version we'll need to
> be careful.
Basically: I'm assuming we will always have users of the old version for
whatever value of old version. To expand on the XI2 case - we changed
the behaviour there based on the version the client announces (XI2.1 or
XI2.2, can't remember) and, a few years later, discovered that the same
display connection was used by different parts supporting different
values. So the toolkit assumed version X, the application assumed
version X+1. Events compatible with X were forwarded to the toolkit, but
if you change the behaviour, it'll break.
>
>> My current idea is to add a wl_pointer.axis_v120 event:
>>
>> wl_pointer.axis_source wheel
>> wl_pointer.axis_v120 30
>> wl_pointer.frame
>> wl_pointer.axis_source wheel
>> wl_pointer.axis_v120 30
>> wl_pointer.frame
>> wl_pointer.axis_source wheel
>> wl_pointer.axis_v120 30
>> wl_pointer.frame
>> wl_pointer.axis_source wheel
>> wl_pointer.axis 10
>> wl_pointer.axis_discrete 1
>> wl_pointer.axis_v120 30
>> wl_pointer.frame
>
> I can think of a few issues with this:
>
> * Can the compositor still have a scroll multiplier setting? Is a 120
> granularity enough? (ie. what happens if the user has a very precise mouse and
> configures a scale multiplier of 0.5?)
> Maybe this value could be a wl_fixed and named e.g. axis_fraction?
The 120 is so deeply embedded in the windows API that I don't think
it'll change anytime soon. And in the kernel now too. 120 is simply a
magic value because it has lots of integer factors, and let's be honest:
a factor of 120 would give you less than a degree granularity on your
average scroll wheel, that's probably enough until mouse wheels go dodo.
That factor is also in the hardware. The highest so far I've seen is 16.
> * What do we do about wl_pointer.axis? Maybe we can just phase it out? I'm not
> a fan of having to support two APIs forever. Plus the current axis event is
> ill-defined.
I don't think we can ever phase out wl_pointer.axis or change its
meaning. All we can do is add to it to reframe its interpretation.
> * Can't we just define a axis_full_turn event sent after binding to wl_pointer,
> which defines what is the amount of axis you need to wait for until you reach
> a full turn? (The axis event already carries a wl_fixed value)
The problem with this is that wl_seat multiplexes pointers. If you have
two mice, the full turn may differ between the mice and you may be at a
fraction of a full turn on any mouse at any time. It simply doesn't
work, sorry (was my first idea too :)
>> Main problems here: this puts a fair bit of implementation requirements on the
>> compositor - skip source/v120/frame for clients < supported version.
>
> Well this is inevitable anyway -- having a new protocol version will require
> compositors to add more version checks.
>
>> wl_pointer.source cannot currently occur on its own, it requires a
>> wl_pointer.axis event in the current protocol. Same with axis_discrete. Empty
>> frames are allowed, I think, but pointless.
>>
>> It also puts a bit of extra logic in the client, you need to remember if you've
>> seen a v120 in this frame and discard the axis/axis_discrete events. If not,
>> then axis events must be processed as normal.
>>
>> A possible workaround would be to add v120 to all axis frames (including
>> touchpad scrolling) but that would then put the compositor in charge of
>> deciding what distance is a wheel click and encodes that information in the
>> protocol.
>
> Side note, the current protocol already requires compositors to decide of an
> axis event value for touchpads.
Not necessarily. libinput at least gives you touchpad events in the same
coordinate space as the input, so forwarding that 1:1 is fine. The
problem is converting that to wheel clicks, but that's done client-side.
>
>> The v120 event must also carry the equivalent to the wl_pointer.axis value so
>> that can be passed on into existing code. So the above would have a v120 event
>> with v120 value of 30 and an axis value of 2.5 (if enforcing 10 units per
>> wheel click).
>>
>> Because the hi-res and lo-res sources are independent, we need to allow for a
>> 0-value v120 event, which basically just tags that frame as
>
> as low-res?
huh, yeah. copy/paste error? but yes, if you have a 0-value v120 event,
that's a marker that "this source has high-res scrolling and if you're
handling the v120, you can ignore the events in this frame"
>
>> The compositor also need to *always* send v120 events for wheel/tilt sources
>> because otherwise the client code becomes even more of a nightmare.
>> Where not backed by libinput the compositor needs to handle that itself.
>
> Agreed.
>
>> Anyway, the below is the best I've come up with so far though I haven't
>> implemented it client-side yet. I'd really appreciate any feedback and/or
>> epiphanies.
>>
>> Cheers,
>> Peter
>>
>> diff --git a/protocol/wayland.xml b/protocol/wayland.xml
>> index b1c930d..cd45407 100644
>> --- a/protocol/wayland.xml
>> +++ b/protocol/wayland.xml
>> @@ -1662,7 +1662,7 @@
>> </request>
>> </interface>
>>
>> - <interface name="wl_seat" version="7">
>> + <interface name="wl_seat" version="8">
>> <description summary="group of input devices">
>> A seat is a group of keyboards, pointer and touch devices. This
>> object is published as a global during start up, or when such a
>> @@ -1771,7 +1771,7 @@
>>
>> </interface>
>>
>> - <interface name="wl_pointer" version="7">
>> + <interface name="wl_pointer" version="8">
>> <description summary="pointer input device">
>> The wl_pointer interface represents one or more input devices,
>> such as mice, which control the pointer location and pointer_focus
>> @@ -2092,9 +2092,97 @@
>> <arg name="axis" type="uint" enum="axis" summary="axis type"/>
>> <arg name="discrete" type="int" summary="number of steps"/>
>> </event>
>
> Nit: we generally add a
>
> <!-- Version 8 additions -->
>
> comment here.
>
>> + <event name="axis_v120" since="8">
>> + <description summary="axis high-resolution scroll event">
>> + Discrete high-resolution scroll information.
>> +
>> + This event carries high-resolution wheel scroll information,
>> + with each multiple of 120 representing one logical scroll step.
>> + For example, a v120 of 30 is one-quarter of a logical
>> + scroll step in the positive direction, the v120 of -60 is
>> + one-half of a logical scroll step in the negative direction.
>> + Clients that rely on discrete scrolling should accumulate the
>> + v120's to multiples of 120 before processing the event.
>> +
>> + This event also carries the axis motion value in the same coordinate
>> + space as the wl_pointer.axis event's value. See the
>> + wl_pointer.axis documentation for details.
>
> Why do we need to duplicate this information?
I'm not 100% convinced on this yet but it allows a compositor to
completely shortcut the other axis handling bits and just deal with the
information in the v120 event. It's not necessary, but it makes the
event more self-contained and the parsing slightly simpler.
I had thought about adding the source to it too but decided against
that, it didn't feel right.
Thanks for the feedback, much appreciated
Cheers,
Peter
>
>> + This event is guaranteed for axis movement from axis sources wheel
>> + and wheel_tilt even if the underlying device does not support
>> + high-resolution scrolling.
>> +
>> + Where a wl_pointer.axis_source event occurs in the same
>> + wl_pointer.frame, the axis source applies to this event.
>> +
>> + Where wl_pointer.axis and/or wl_pointer.axis_discrete events
>> + occur in the same wl_pointer.frame, those events reflect an
>> + accumulated scroll amount. Clients handling axis_v120 events can
>> + usually ignore the axis and axis_discrete events.
>> +
>> + For example, a wheel generating events every 25% of each physical
>> + wheel detent may create the following event sequence:
>> +
>> + wl_pointer.axis_source wheel
>> + wl_pointer.axis_v120 30
>> + wl_pointer.frame
>> + wl_pointer.axis_source wheel
>> + wl_pointer.axis_v120 30
>> + wl_pointer.frame
>> + wl_pointer.axis_source wheel
>> + wl_pointer.axis_v120 30
>> + wl_pointer.frame
>> + wl_pointer.axis_source wheel
>> + wl_pointer.axis_v120 30
>> + wl_pointer.axis 10
>> + wl_pointer.axis_discrete 1
>> + wl_pointer.frame
>> +
>> + For backwards compatibility, the wl_pointer.axis event for a wheel
>> + or wheel tilt must contain the value of the full wheel movement. It
>> + must not be sent for a fraction of a physical detent.
>> +
>> + For backwards compatibility, the wl_pointer.discrete event for a wheel
>> + must contain the value of the full wheel movement. It must not be
>> + sent for a fraction of a physical detent and it must not be 0.
>> +
>> + For backwards compatibility, the wl_pointer.axis_source event in a
>> + frame containing only a wl_pointer.axis_v120 event must not be sent
>> + to clients that do not support wl_pointer version 8 or later. In
>> + other words, the above sequence to such clients is reduced to:
>> + wl_pointer.axis_source wheel
>> + wl_pointer.axis 10
>> + wl_pointer.axis_discrete 1
>> + wl_pointer.frame
>> +
>> + Clients must treat wl_pointer.axis_v120 and
>> + wl_pointer.axis/axis_discrete as two independent event sources.
>> + There is no guarantee that the steps indicated by the v120 events
>> + match the steps by wl_pointer.axis_discrete events. In other words,
>> + an accumulated v120 value of 360 does not guarantee an accumulated
>> + axis_discrete value of 3.
>> +
>> + The value of a wl_pointer.axis_v120 event may be 0.
>> + This is the case where a low-resolution event has no equivalent
>> + high-resolution event. The value of 0 indicates that any
>> + wl_pointer.axis/axis_discrete events in the same event can be
>> + ignored.
>> +
>> + The axis motion's coordinate space is identical to the
>> + wl_pointer.axis event.
>> +
>> + The order of wl_pointer.axis_v120 and wl_pointer.axis_source is
>> + not guaranteed.
>> + </description>
>> + <arg name="time" type="uint" summary="timestamp with millisecond granularity"/>
>> + <arg name="axis" type="uint" enum="axis" summary="axis type"/>
>> + <arg name="motion" type="fixed" summary="length of vector in surface-local coordinate space"/>
>> + <arg name="v120" type="int" summary="scroll distance as fraction of 120"/>
>> + </event>
>> </interface>
>>
>> - <interface name="wl_keyboard" version="7">
>> + <interface name="wl_keyboard" version="8">
>> <description summary="keyboard input device">
>> The wl_keyboard interface represents one or more keyboards
>> associated with a seat.
>> @@ -2208,7 +2296,7 @@
>> </event>
>> </interface>
>>
>> - <interface name="wl_touch" version="7">
>> + <interface name="wl_touch" version="8">
>> <description summary="touchscreen input device">
>> The wl_touch interface represents a touchscreen
>> associated with a seat.
>>
>> _______________________________________________
>> wayland-devel mailing list
>> wayland-devel at lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/wayland-devel
More information about the wayland-devel
mailing list