Some of my thoughts on input for wayland

[Kristian Høgsberg]

On Mon, Jan 24, 2011 at 2:41 PM, Marty Jack <martyj19 at comcast.net> wrote:
>
>
> On 01/24/2011 02:22 PM, Chase Douglas wrote:
>> On 01/24/2011 01:57 PM, Jesse Barnes wrote:
>>> On Sun, 23 Jan 2011 21:03:58 -0500
>>> Chase Douglas <chase.douglas at canonical.com> wrote:
>>>> First I'd like to address what I think we can learn from X. X11 has a
>>>> core protocol and an XInput extension with two major versions. To
>>>> develop additions to the input system in X you must meet two obligations:
>>>>
>>>> 1. Develop alongside all the other work going on in X
>>>> 2. Be backwards compatible with the previous input systems
>>>> 3. Be integrated into the same display server source code
>>>>
>>>> I think we could take a different approach with Wayland: separate input
>>>> from display. What does the input system need from the rest of X?
>>>> Nothing really other than window regions and hierarchy on the screen.
>>>
>>> Yes, but in that small statement "window regions and hierarchy" lie
>>> dragons. :)
>>>
>>> One of the main reasons we don't have input transformation in X is due
>>> to this very problem.  Having the compositor separate from the server,
>>> with all that implies (split input handling, duplicated window
>>> hierarchy, etc.) makes doing transformed input very difficult (so much
>>> so that no one's really tried).
>>>
>>> Keeping input and display together makes this much easier.
>>>
>>> If they were split out for Wayland, the input process would have to
>>> communicate with the display server, and duplicate window region data
>>> (including transforms), and also handle potential races between input
>>> handling and display updates (animations and the like).  So we'd end up
>>> with a potentially very complex inter-process protocol.
>>>
>>> So overall, I think splitting input and display would create more
>>> problems than it would solve.  I largely agree with your goals though:
>>> you want it to be as simple and easy to change as possible, I just
>>> don't think splitting it out as radically as you propose would achieve
>>> that.
>>
>> I'm not extremely well versed in multi-process communication, but I was
>> thinking we may be able to do something like rcu in the kernel, but
>> apply it to the compositor window hierarchy. Keep the window hierarchy
>> in shared memory with the input server. Then wrap the IPC mechanism into
>> an easy to use shared library for both wayland and input server sides.
>> The compositor would be the only process with write capabilities to the
>> memory region, so it would never have to wait for locking. The input
>> server(s) should be fast since most input isn't occurring while windows
>> are moving (I think :).
>>
>> This would be somewhat complex in implementation, but if it's a shared
>> library then the complexity is hidden away. The ultimate question in my
>> mind is: will this be performant enough?
>>
>> (And if it's really obvious that I don't have a clue what I'm talking
>> about with regards to shared data and IPC, please flame gently :)
>>
>>>> As input devices are advanced into true 3 dimensional space, we may find
>>>> a need for even more intricate input service mechanisms. A more future
>>>> proof model may involve the ability to dynamically slot in input systems
>>>> as plugins. In this way, we might also be able to deprecate older input
>>>> protocols over time.
>>>
>>> I heard something like that was proposed at the recent Khronos meeting;
>>> it sounded like STREAMS but for input, maybe krh can point you at the
>>> proceedings or something, it seemed like a potentially reasonable way
>>> of handling the variety of input needs we have out there today, while
>>> preserving flexibility for the future.
>>
>> Interesting... I would be grateful if someone could find it.
>>
>> Thanks!
>>
>> -- Chase
>> _______________________________________________
>> wayland-devel mailing list
>> wayland-devel at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/wayland-devel
>>
>
> Let's not design a complex solution to a simple problem before we have it working one way.  Then if it doesn't perform properly, you rewrite it one more time now that you understand it and add complexity.
>
> + The compositor would be the only process with write capabilities to the
> + memory region, so it would never have to wait for locking.
>
> This isn't correct.  The compositor would have to wait to write whenever any other process had the lock, which it would need to hold in order to read.  And, if the input side is going to do anything useful, it has to write what happened somewhere, or it may as well not have run at all.
>
> + This would be somewhat complex in implementation, but if it's a shared
> + library then the complexity is hidden away. The ultimate question in my
>
> This isn't correct.  The complexity is right there being maintained by the maintainer of whatever piece of software you decide to package the complexity into.  And you want as many people as possible to be able to understand it so as many people as possible can contribute to the development organization.
>
> The main reason why you don't need a separate thread to handle the input is that it is sleeping 99.9% of the time waiting for the user to do something and then it runs in linear time.  It would be fine in whatever thread the compositor is blocking on its poll.
>
> Not to mention Kristian has done a nice job of pointing out that it's a lot of shared state.

I think a lot of the focus on "input in a thread/process" or
low-latency input comes from X limitations, where the X server does a
lot of work on behalf of the clients (rendering, loading fonts, etc).
And even there, it doesn't matter too much (except for cursor motion),
since the events batch up and the X server and clients can do motion
event compression and deal with it.  In Wayland, the compositor only
tweaks the scene graph to mark a region dirty or move a window.  The
only real work the compositor ever does is composite the desktop, at
most once per frame and that doesn't take a lot of Wayland CPU time
(and typically not a lot of GPU time either).  In the drm backend, we
don't even block in eglSwapBuffer, the "page flip done" event is
delivered asynchronously.  So, unlike X, which can be busy doing any
kind of job for a client, the Wayland compositor is almost always idle
waiting to handle input events.

Kristian