Atomic KMS API lacks the ability to set cursor hot-spot coordinates

[Pekka Paalanen]

On Thu, 19 Mar 2020 15:30:03 +0100
Hans de Goede <hdegoede at redhat.com> wrote:

> Hi,
> 
> On 3/19/20 1:58 PM, Pekka Paalanen wrote:
> > On Thu, 19 Mar 2020 12:49:27 +0100
> > Hans de Goede <hdegoede at redhat.com> wrote:
> >   
> >> Hi,
> >>
> >> On 3/19/20 11:00 AM, Pekka Paalanen wrote:  
> >>> On Wed, 18 Mar 2020 15:28:02 +0100
> >>> Hans de Goede <hdegoede at redhat.com> wrote:
> >>>      
> >>>> ATM the Atomic KMS API lacks the ability to set cursor hot-spot
> >>>> coordinates. Mutter (and Weston) have tried to emulate this by shifting
> >>>> the coordinates for where to draw the cursor by the hotspot-coordinates
> >>>> and always using 0x0 for the hotspot.
> >>>>
> >>>> But this breaks the so called "seamless mouse mode" for virtual-machines
> >>>> and there really is no way to fix this but to allow passing the proper
> >>>> hotspot coordinates to the virtual gfx-card.
> >>>>
> >>>> Seamless-mode consists of 2 parts:
> >>>>
> >>>> 1) Letting the VM-viewer window-system draw the cursor as it normally
> >>>> would draw it.
> >>>>
> >>>> 2) Giving absolute coordinates of the mouse to the VM by e.g. emulating
> >>>> an USB drawing tablet. These coordinates come from translating the
> >>>> coordinates where the VM-viewer window-system is drawing the cursor
> >>>> to an absolute position using the top left corner of the view as 0x0
> >>>> and the bottom right corner as max-abs-x,max-abs-y.  
> >>>
> >>> Hi,
> >>>
> >>> is the VM-viewer then hiding or autonomously moving what the display
> >>> server inside VM has put on the KMS cursor plane?  
> >>
> >> Yes and no, it is not the VM-viewer which is hiding what the
> >> display-server inside the VM has put on the KMS cursor plane,
> >> the VM-viewer negotiates seamless mouse mode with the hypervisor
> >> and then the hypervisor just ignores the cursor-plane except for
> >> sending "sprite" changes to the VM-viewer.  
> > 
> > Hi,
> > 
> > I don't think I understand what you're saying, but I assume that I was
> > right in that the VM cursor plane content will not be shown always
> > exactly in the very position the compositor inside the VM puts it.  
> 
> Right, when seamless mouse mode is enabled the VM's cursor plane will
> not be shown *at all*, instead the VM-viewer's window-system's
> cursor plane is shown.
> 
> Sprite changes (normal cursor, caret, window-resize cursor, etc.)
> are forwarded from the VM-s cursor-plane to the VM-viewer
> which then requests that as cursor to the window-system under
> which the VM-viewer is running.
> 
> > Maybe the example further below explain the issue I envision.
> >   
> >>> If so, sounds like hilarity would ensue with Weston.
> >>>
> >>> Weston does not actually know what a cursor is. Weston will happily put
> >>> any arbitrary non-cursor content onto the KMS cursor plane if it happens
> >>> to fit. It's just that usually there is a small surface top-most that
> >>> ends up on the cursor plane, and that surface accidentally happens to
> >>> be a cursor by Wayland protocol.
> >>>
> >>> It's not difficult to get e.g. weston-simple-shm window to be shown on
> >>> the KMS cursor plane: just hide the real cursor from the client.
> >>>
> >>> No, it's not an oversight. It is called "making maximal use of the
> >>> available KMS resources" by using KMS planes as much as possible to
> >>> avoid compositing by rendering with Pixman or OpenGL.  
> >>
> >> Yes it sounds like this will break with Weston, note that it already
> >> is broken in Weston, if you run e.g. Fedora 32 beta + its Weston
> >> package inside a VirtualBox VM then start gnome-terminal (so
> >> that you get a caret cursor instead of the default one) and try to
> >> select text. This will result in the wrong text being displayed
> >> because Weston does not relay cursor hotspot info to the GPU,
> >> also see:
> >> https://gitlab.gnome.org/GNOME/mutter/issues/1094
> >>
> >> Where the symptoms of this are described in more detail
> >> (they are identical for Weston and mutter).  
> > 
> > Right, that's the problem with the hotspot.  
> 
> Ack.
> 
> >> Fixing this will require the discussed KMS atomic API changes
> >> and also changes on the Weston and mutter side to pass through
> >> the hotspot info.  
> > 
> > The problem I am referring to is that to the user looking at the
> > VM-viewer, suddenly an arbitrary application window (e.g.
> > weston-simple-shm) starts to act as if it was the cursor, when there is
> > no real cursor shown. You have a random window unexpectedly moving
> > around, as if you had started dragging it around with your mouse.  
> 
> Correct.
> > The only way to fix that is to stop Weston from putting non-cursor
> > content on the cursor plane.  
> 
> Correct.

Is that something that should be done?

If the hotspot property also had a "disabled" value, then Weston could
set the hotspot to disabled when it is using the cursor plane for
non-cursor content and not lose the feature. And of course set hotspot
correctly when it in fact is a cursor (but for what input?).

> 
> > It sounds like your VM-viewer makes the assumption that the pointer
> > input device it delivers to the VM is the one that will control the KMS
> > cursor plane position inside the VM. Is that right?  
> 
> Correct.
> 
> > What if the desktop inside the VM is controlled by a remote, e.g. VNC?
> > Then the input events to the VM are completely unrelated to the
> > expected motion of the cursor. Do you just tell the users to stop using
> > the seamless mode in that case?  
> 
> A VNC viewer (which is not using seamless mode itself) has the same issue
> of a mismatch between the cursor position of the window-system it is a client
> of and the cursor position of the window system inside the VM.
> 
> A VNC viewer typically works around this by changing the window-system
> cursor to transparent and drawing its own cursor, the transparent sprite
> or disabling of the cursor-plane will get forwarded to the window-system
> under which the VM-viewer runs so this will work fine.
> 
> A real problem though is the absolute input mode, when e.g. the VNC
> viewer is not using something like seamless mouse mode it will want
> to do the usual (nasty) tricks of confining the pointer to the window
> and warping it to the center after each pointer move so that mouse
> can be moved "endlessly' inside the window. This requires relative
> input, so emulating a mouse instead of say a drawing tablet, which
> breaks seamless mode.
> 
> So although the way the cursor is drawn typically does not require
> disabling seamless mode, the input problems do require disabling
> seamless mode.
> 
> In my experience (I have worked on VDI (spice) a couple of years),
> running a VM-viewer inside a VM is something which not a lot of
> people do. I guess the fact that this works quite poorly might
> be one of the reasons people do not do this.

Sorry. I meant that you run a desktop in a VM. Then you control that
desktop from some remote via VNC, and you still look at a VM's
VM-viewer.

You can replace VNC with any remote input thing. E.g. USB devices
plumbed directly through to the VM while still looking at the VM-viewer
for output.

Niche? Yes, quite likely.

> But all VMs I'm familiar with allow disabling seamless mode for
> compatibility with corner cases like this. This does lead to a
> seriously degraded user-experience though. For just using an
> office suit or browser inside a VM seamless mode really is a
> lot more pleasant for the user.  I still remember when seamless
> mode first got introduced, in the beginning it used to not
> always work and fixing the fallback to the confine + warp tricks
> then always was a serious itch which I tried to scratch quickly.
> Breaking seamless mode also is a good way to quickly accumulate
> a lot of bug reports from end users in my experience.

Btw. the warp trick is purely a legacy X11 thing, we have relative
motion interfaces nowadays at least in Wayland and I think also with
XI2?

But if you need to grab the pointer to allow endless relative motion,
then you grab, that hasn't changed from end user perspective.

I think one of the major reasons why Wayland pointer relative motion
and confinement extensions were designed was VM- etc. viewers, and of
course games.

> > What if display servers stop using the cursor plane completely, because
> > people may hit a case where a VM-viewer makes the wrong assumption about
> > which input device is associated to which cursor plane inside the VM?  
> 
> The confine + warp trick is typically the default mode and only
> if the guest indicates through e.g. a guest-agent process that
> seamless mode is supported then seamless mode is enabled.
> 
> IOW the VM is careful to not enable it when it might not work.

How would the guest-agent know? Does it check that there is literally
only one pointer input device and that comes from the VM-viewer?
Or does it limit seamless to white-listed display servers perhaps?

> OTOH most distros now ship with the necessary agents installed
> by default, so e.g. a F32 beta install will automatically use
> seamless mode under QEMU/KVM (spice display protocol) or
> VirtualBox.  If a user uses say F32 + a vnc-viewer inside it
> which needs to do confine + warp tricks then the user will have
> to disable seamless mode manually. In my prior experience
> working on VDI this is usually something which users are
> understanding of and not a problem.

Yeah, making the association between the input and the cursor plane
is unsolved.


Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/dri-devel/attachments/20200319/d3ec2c3f/attachment-0001.sig>