Remote desktop, keyboard auto-repeat, and network jitter

[Erik Jensen]

(Note: to try to avoid overloading the word "client", I refer to the
wayland client as "the application" and explicitly say "the remote
desktop client" to refer to the machine the user is connecting from.)

They Wayland protocol currently delegates handling of keyboard
auto-repeat to the application:

 * The compositor informs the application of the user's current repeat
settings via wl_keyboard::repeat_info.
 * The compositor sends one key-pressed event when the key is first
pressed, and a key-released event when it is ultimately released.
 * The application is responsible for applying the settings from
repeat_info to generate repeated characters for the held key, if
appropriate.

This works great locally, and gives the application flexibility to
handle a held key however makes sense for the context. Unfortunately,
it can run into trouble in the context of a remote desktop session.

When connecting to a remote machine over the internet, it's not
uncommon to experience a network hiccup of a second or more,
especially if connecting via a less-reliable medium (cellular, weak
wifi, satellite, et cetera). This can result in a key up event being
delayed with respect to its associated key down event, leading to
undesired and annoying (and occasionally dangerous) key repeats.

Currently, the only solution is to turn off key repeat. This is safer,
but obviously has a usability impact. The remote desktop tool might
choose to work around the usability impact by synthesizing a key up +
key down pair of events every time the key is auto-repeated on the
remote desktop client, but this has the issue of making it impossible
to actually hold down an auto-repeating key.

Ideally, it would be possible to guarantee that a key will be repeated
only once the user has actually held down the key a sufficient length
of time on the remote desktop client, not subject to network jitter.
Leaving aside remote desktop tool <-> compositor communication for the
moment (likely a topic for a different list), I can think of a couple
of different approaches to adjusting the compositor -> application
protocol to enable this.

Idea 1: Signal the application to start repeating the
most-recently-pressed key, if applicable.

Repeat would initially be disabled. The remote desktop tool would
inform the compositor when the user had been holding a key down for
longer than the delay threshold, as measured on the remote desktop
client (removing any network delay issues), at which point the
compositor would inform the application that it should commence
repeating.

This might be achieved by specifying that a new repeat_info event
should be applied immediately to the most recently-pressed,
currently-held key, if any (though this is not how toolkits currently
handle it), or by specifying a new event type.

This approach will at least ensure that the user intended to start
repeating, but could still potentially result in more repeats than
intended. E.g., if the video stream freezes due to a network hiccup,
the user might immediately release the key to try to avoid too many
repetitions, but the application would keep generating repeats until
the key up event was received.

Idea 2: Send explicit individual repeat events to the application.

Both the delay and repeat rate would be applied on the client machine,
ensuring that the number of repeats is commensurate with the actual
length of time the user held down the key, regardless of network
conditions. The remote desktop tool would generate a key-down event,
some number of key-repeat events, and a key-up event.

Via the Wayland protocol, the application would receive a repeat_info
event signaling no auto repeat (rate of zero), or perhaps a special
value signalling that repeats will be provided by the compositor. It
would then receive a key-pressed event, some number of key events with
a new "repeat" key_state, followed by a key-released event. The
application would be free to process or ignore the repeat events
depending on the current context.

I would lean toward this approach, barring some technical reason it
wouldn't work well in practice.


Would either of these approaches be tenable? Obviously, they both
require updates to toolkits, but I don't think that's avoidable. Older
applications would fall back to having repeat disabled when using a
remote desktop tool, which might be annoying until they are updated,
but would at least be safe. Is there any other solution I'm
overlooking?

Thanks!
Erik