[PATCH v4 2/8] drm/atomic: Add support for mouse hotspots

[Pekka Paalanen]

On Fri, 7 Jul 2023 13:54:21 -0700
Michael Banack <banackm at vmware.com> wrote:

> On 7/7/23 01:38, Pekka Paalanen wrote:

...

> >>>>>>> The question of which input device corresponds to which cursor plane
> >>>>>>> might be good to answer too. I presume the VM runner is configured to
> >>>>>>> expose exactly one of each, so there can be only one association?  
> >>>>>> As far as I know, all of the VM consoles are written as though they
> >>>>>> taking the place of what would the the physical monitors and input
> >>>>>> devices on a native machine.  So they assume that there is one user,
> >>>>>> sitting in front of one console, and all monitors/input devices are
> >>>>>> being used by that user.  
> >>>>> Ok, but having a single user does not mean that there cannot be
> >>>>> multiple independent pointers, especially on Wayland. The same with
> >>>>> keyboards.  
> >>>> True, and if the userspace is doing anything complicated here, the
> >>>> hypervisor has to be responsible for ensuring that whatever it's doing
> >>>> works with that, or else this system won't work.  I don't know that the
> >>>> kernel is in a good position to police that.  
> >>> What do you mean by policing here?
> >>>
> >>> Isn't it the hypervisor that determines what virtual input devices will
> >>> be available to the guest OS? Therefore, the hypervisor is in a
> >>> position to expose exactly one pointer device and exactly one
> >>> cursor plane to guest OS which means the guest OS cannot get the
> >>> association wrong. If that's the general and expected hypervisor
> >>> policy, then there is no need to design explicit device association in
> >>> the guest kernel UAPI. If so, I'd like it to be mentioned in the kernel
> >>> docs, too.  
> >> I'm not entirely sure how to fit what you're calling a "pointer" into my
> >> mental model of what the hypervisor is doing...  
> > My definition: A pointer is a pointing input device that requires a
> > cursor image to be drawn at the right location for it to be usable.  
> Right, but normal desktops (and our consoles) expect multiple input 
> devices to feed into a single cursor.  So the connection between the 
> on-screen cursor and the corresponding input-devices is not clear to me 
> when you start talking about multiple pointers, even without any 
> hypervisors in the picture.

The connection is simple: there is an independent on-screen cursor for
each logical pointer. How that cursor is drawn is irrelevant to the end
user, and Wayland compositors (a type of a display server) will use any
means necessary to draw it.

Each logical pointer has one cursor that is independent from all other
logical pointers. Each logical pointer can have any number of input
devices controlling it. The assignments are decided by the userspace
and implemented in a display (window system) server.

This has been ingrained into the fundamental design of Wayland, even if
the feature is rarely used in practise. The window system may expose
multiple independent pointers to applications, and each pointer may also
interact with the same window simultaneously. This naturally leads to
the question "which cursor goes with which input device?", and the
kernel, or anything below it, does not know that if there are multiple
possibilities.


> >> So I guess I'm not clear on what kind of usermode<=>kernel contract you
> >> want here if the kernel isn't what's owning the translation between the
> >> mouse input and the cursor position.  The hypervisor awkwardly has to
> >> straddle both the input/graphics domain, and we do so by making
> >> assumptions about how the user desktop is going to behave.  
> > Correct. In order to reduce that awkwardness, I encourage you to write
> > down the expectations and requirements in this new Linux UAPI (the KMS
> > cursor place hotspot property). Then you can be much more confident on
> > how a random Linux desktop will behave.
> >
> > It will also help the reviewers here to understand what the new UAPI is
> > and how it is supposed to work.  
> 
> The cursor hotspot is I think fairly straightforward, as far as what 
> that means for how hypervisor clients are expected to draw the mouse, 
> and Zack's working on that part.
> 
> My point was that how the hypervisor then sends input is sort of outside 
> the scope of the graphics portion here, and I think probably outside the 
> current Linux UAPI entirely (unless there's some other input/topology 
> system somewhere else I'm not familiar with).

I would not say that the hotspot property is in any way obvious. I've
spent my whole Wayland and compositor developer career of over 10 years
never seeing the need of the kernel knowing the hotspot myself, because
I never use VMWare like tools.

You cannot describe why hotspot property is needed or how it works
without explaining the input side. The hotspot property is actually
really weird, because its existence requires combining the input system
with the graphics system for it to make sense. The input system used to
be out of scope indeed, but this addition forces it in scope. This is
the first time that I know of when the kernel or components below the
kernel needs to know, hence there is no existing infrastructure in
Linux to record that topology or anything.

(Sidetrack: for ultra-fast displays, say 1000 Hz refresh for seamless
hand-eye coordination, it could make very much sense for userspace to
off-load arbitrary KMS plane positioning through the kernel into
hardware in general, tying the plane position to some input device
position temporarily. It might make some sense even with today's
consumer hardware. So I don't think the concept is fundamentally
limited to paravirtualized environments. But going there would need a
lot more work than I am willing to suggest to VMWare to tackle to just
make their own products better.)

> > However, in your case, the userspace (Wayland or X11 display server) is
> > not free to choose any arbitrary input-cursor association they might
> > want. You have a specific expectation that all pointing devices control
> > the same pointer. Since the hotspot property is exclusive to your use
> > case, I think it make sense to write down the expectations with the
> > hotspot property. Guest userspace has to explicitly program for the
> > hotspot property anyway, so it can also take care of your requirements.  
> I see your point, and I can see the value in documenting something to 
> that effect, if only because it's /useful/ for userspaces trying to work 
> with this.  (And the only way anyone is using this today.)
> 
> But I'm still a little fuzzy on what portion of that is within the Linux 
> UAPI scope for the hotspot...
> 
> It seems like it might be more useful to restrict the scope of the Linux 
> UAPI contract for how the hotspot property works to just how userspace 
> can expect the hypervisors to display it on screen, and not try to tie 
> in any expectations for how mouse input is going to work.
> 
> Like, VMware is using virtual mouse devices here, but another hypervisor 
> might have no kernel mouse device and instead inject input entirely 
> through a userspace daemon?  So even trying to express the input part of 
> the contract in terms of kernel input devices I'm not sure makes sense.
> 
> I guess my fear is that I don't want to lock this down in a way that 
> excludes some well-meaning hypervisor that implements the graphics part 
> correctly but does something just weird enough on the input side to not 
> be considered compliant.
> 
> So I guess I would vote for trying to include something to that effect 
> as context or explanation, but not try to strictly define how that works?

Yes, exactly.


Thanks,
pq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/dri-devel/attachments/20230710/3edc73d7/attachment.sig>