Some of my thoughts on input for wayland

[Chase Douglas]

On 01/24/2011 05:36 AM, Marty Jack wrote:
> On 01/23/2011 09:03 PM, Chase Douglas wrote:
>> Hi all,
>>
>> I haven't been involved in wayland development at all yet :), but I have
>> been working on XInput 2.1 multitouch additions and gesture work in
>> Ubuntu. I have a few thoughts on how a new input system for wayland
>> might work.
>>
>> To go along with that, I have no idea if these ideas have been discussed
>> before or not, nor whether the wayland architecture would allow them.
>> These are just some generic thoughts I've had on an input service
>> architecture.
>>
>> First I'd like to address what I think we can learn from X. X11 has a
>> core protocol and an XInput extension with two major versions. To
>> develop additions to the input system in X you must meet two obligations:
>>
>> 1. Develop alongside all the other work going on in X
>> 2. Be backwards compatible with the previous input systems
>> 3. Be integrated into the same display server source code
>>
>> I think we could take a different approach with Wayland: separate input
>> from display. What does the input system need from the rest of X?
>> Nothing really other than window regions and hierarchy on the screen.
>>
>> My proposal would be to create a new input system project (inland?) and
>> define a standard of access between wayland and the new input system.
>> The access could be provided through shared memory or some other low
>> latency IPC. This would allow mixing and matching of display servers and
>> input servers, and separate out the development practices and timelines
>> for greater flexibility.
>>
>> Now, this may seem crazy at first. Once a new input system is defined,
>> wouldn't we want to standardise upon it and not change it? Hopefully
>> yes. But think of wayland itself. It was conceived mostly due to issues
>> with X graphics. Imagine if 10 years from now the graphics stack needed
>> another rewrite, but the input stack was perfectly fine. We could
>> transition to a new graphics server without modifying any application
>> input code if the input system was easily transferable.
>>
>> There's another issue I've noticed as we've gone through XI 2.1 and
>> gesture work: We are building multiple serial event delivery mechanisms.
>> Here's an example:
>>
>> 1. Touchscreen nput comes into X from Linux evdev interface
>> 2. XI 2.1 touch events are generated
>> 3. uTouch gesture recognizer receives events through passive grab on
>> root window
>> 4. Gesture recognizer recognizes a gesture
>> 5. No client is subscribed for the gesture
>> 6. Gesture recognizer relinquishes touch grabs
>> 7. Touches propagate through X server
>> 8. No other clients are found for touches
>> 9. One of the touches is turned into pointer emulation
>> 10. Pointer events propagate through X server once more
>>
>> We see there are three event delivery mechanisms: gestures, touches, and
>> pointer emulation. In each case, we are potentially transforming the raw
>> input data into a different format and looking for clients who selected
>> for appropriate event types. There's also a defined precedence ordering:
>> gestures -> touches -> pointer emulation.
>>
>> One could argue whether this is a proper precedence ordering or not, but
>> the point is that there is a precedence ordering. In fact, I wonder if a
>> better ordering might be:
>>
>> gesture grabs -> touch grabs -> pointer grabs -> gesture selections ->
>> touch selections -> pointer selections.
>>
>> As input devices are advanced into true 3 dimensional space, we may find
>> a need for even more intricate input service mechanisms. A more future
>> proof model may involve the ability to dynamically slot in input systems
>> as plugins. In this way, we might also be able to deprecate older input
>> protocols over time.
> 
> You don't appear to yet be considering the keyboard side of things, which is equally important and since they interact due to accessibility, should be designed together.  I would point out that input methods appear to be a lot like pointer gestures, in that several user actions map to one thing, and could probably be handled very similarly.

You're right here :). I have never touched X keyboard code, so I didn't
really mention it. Would keyboard handling fit poorly into a modular
input system? Is there some interaction with the output display server
that requires more interconnect than I wrote above?

> I don't think Kristian has quite the same view of grabs that you do, but I don't want to misrepresent his thinking on it, so he can talk about that himself if he chooses.

Heh, I've heard his distaste of grabs. I haven't had a chance to
research other mechanisms, so I'm just going on what I know through X.
I'm certainly not opposed to a better mechanism :).

> I do hope we can do this in a way that doesn't result in yet another overhead process.

Can you describe more here? If you are meaning a real computer
"process", I think it would help to split input and output display into
separate processes. Practically every system shipping in the future will
have multiple cores, so making the input system wait around while output
does its thing seems unnecessary.

Thanks for the feedback!

-- Chase