Wayland Relative Pointer API Progress

[x414e54]

On Tue, Jun 30, 2015 at 4:22 AM, Bill Spitzak <spitzak at gmail.com> wrote:
>
>
> On Sun, Jun 28, 2015 at 7:32 AM, x414e54 <x414e54 at linux.com> wrote:
>>
>>
>> Clients do not draw the mouse cursor, either the GPU (using hardware
>> overlays) or the WM does.
>
>
> Yes, I want to allow clients to use this CPU/WM support. Currently the
> client *has* to draw the cursor (and hide the hardware one) to sync it's
> position with the drawing. If instead the client just *moves* the cursor,
> then the cursor is actually drawn by the compositor like you say.
>

Okay lets say for whatever reason you allow the client direct access
to set the cursor position in a way that did not introduce any extra
lag.

Lets say the client gets direct access to the GPU to set a hardware
overlay position (I think hardware cursors are currently disabled due
to some intel vsync issue but this is does not effect the point).

Completely ignoring any security issues with direct access and any
translation issues from transformed or scaled windows or mice which do
not move in two dimensions.

You STILL cannot sync a scrollbar with the mouse position because the
event loop thread should not be tied to the render loop thread and the
call to swap buffers would be out of sync with the call to set
position.

Even if you decided putting it all onto one thread or waited you would
still have have issues where vsync refreshed the cursor position
before the window or the processes was preempted. As the calls are not
atomic.

As there is no specific egl api for cursors anyway you would have to
drop hardware cursors and draw your own software cursor on top of your
surface.

Either way your event loop is now completely tied into your render
loop, meaning your render loop will also introduce extra latency to
your event loop. Lets be extreme and say the render loop updates at
1fps on my computer (because it is really slow, something out of your
application's control).
The mouse cursor now moves at 1fps and every-time I click nothing
happens for a whole second.

How is this "smooth"?

I really could not trust that any application would be bothered to
implement this correctly.

>>
>> It is a bit of an extreme case example but:
>>
>> 1) User moves mouse and WM adds to clients queue then blocks mouse
>> movement.
>> 2) The application's render thread hangs but event thread continues.
>> 3) Mouse does not move.
>
>
> Lets add the next steps:
>
>  4) Compositor does not get a cursor position or pong request in the next
> 1/10 second or so.
>  5) Compositor draws the cursor in the new position just like it does now
>
> I am just trying to allow clients to take advantage of sync if they are
> capable of it. Falling back to the previous behaviour is acceptable. In fact
> this is better than clients hiding the cursor and drawing their own in that
> such fallbacks work. A frozen client drawing it's own cursor means the
> cursor will appear only when the user moves it out of the window, and at a
> pretty unpredictable place, and the old cursor image will still be there on
> screen.

An event loop in most applications is separate from the render loop so
you cannot use ping/pong events. The main point of the ping/pong is to
pop up the force quit dialog if the application stops processing input
(because the user would be unable to click close), not pauses in
rendering output.

You also cannot use an update to cursor position because the cursor
genuinely may not be moving. The compositor may be able to see the
relative motion event but it does not know if the client wants to move
the cursor or not.

Also you cannot use a buffer update because the applications should
not be required to commit a buffer that has not changed just to
move/not move the cursor.

Furthermore you are putting too much trust into a perfectly performing
client which are much harder to implement in practice than theory.

>
>> There should be no difference between when moving a window and not
>> because the client is not involved directly.
>
>
> Whether a client is involved "directly" should not be visible to a user.

I am really not sure you understand how computers work. Are you some
how under the impression that latency does not exist?

Some older systems the client probably is involved in moving a window.
If they do not have double buffering and draw directly to the onscreen
frame-buffer for example.

The choice here is either wait for the client and have laggy window
moving or just draw whatever junk was in the framebuffer. I guess an
example of this on older oses was when you moved the window and ended
up with the decoration or the window above rendered partially inside a
hung application.

That would be even worse in your scenario because if the application
below lagged then the window on top would not be allowed to move until
the application below timed out.

Either way it is pretty obvious to the user that the client is
involved directly.

> I'm sure this works this way on Windows due to simplistic implementation of
> the compositor, and that is why resizing does not work this way (as you
> noticed). However that does not negate the fact that this makes the window
> positioning look better, even on "underpowered" computers. What I am
> proposing is that clients be able to take advantage of this to make other
> interactions besides window dragging be perceptually smoother.

Resizing works that way because reallocating buffers on the GPU is an
expensive and slow operation. The client then has to update and redraw
which probably takes less time than resizing the buffer but it can
still take longer than just a normal draw as it may have to
recalculate layouts.

The benefit Wayland has is that it will not update the previous image
on screen until the new buffer is filled and attached which removes
the garbage or black or white area on resizing. This is what the
"every frame is perfect" is about not about syncing a cursor to
scrollbars.

Take a finger or pen for example how is that ever going to be in sync
with moving a window? In future the majority of users will not be
using a mouse to interact with a window.

>
>> For native GUIs Windows uses a completely different system of drawing
>> based on WM_PAINT events. I believe even scrollbars are drawn and
>> moved server side (by the WM), clients do not render scrollbars
>> themselves. This would be similar to the (over the top) subsurface
>> style approach I suggested earlier.
>
>
> No that is not how it works. All rendering is done on the client side in
> modern Windows. It is done by system libraries but they are running in the
> client process and drawing into a memory-mapped image just like Wayland
> does.
>

Would be nice for some evidence of this before making such a claim.

Regardless of if the actual theme is draw client side it does not mean
it is rendering to only one surface/view/texture. There can be many
individual subsurfaces or views that are part of the window, which may
be clipped or moved by the compositor or GPU without the application
needing to be involved.