[PATCH v4 2/8] drm/atomic: Add support for mouse hotspots

[Michael Banack]

On 7/10/23 01:17, Pekka Paalanen wrote:
> On Fri, 7 Jul 2023 13:54:21 -0700
> Michael Banack <banackm at vmware.com> wrote:
>
>> On 7/7/23 01:38, Pekka Paalanen wrote:
> ...
>
>>>>>>>>> The question of which input device corresponds to which cursor plane
>>>>>>>>> might be good to answer too. I presume the VM runner is configured to
>>>>>>>>> expose exactly one of each, so there can be only one association?
>>>>>>>> As far as I know, all of the VM consoles are written as though they
>>>>>>>> taking the place of what would the the physical monitors and input
>>>>>>>> devices on a native machine.  So they assume that there is one user,
>>>>>>>> sitting in front of one console, and all monitors/input devices are
>>>>>>>> being used by that user.
>>>>>>> Ok, but having a single user does not mean that there cannot be
>>>>>>> multiple independent pointers, especially on Wayland. The same with
>>>>>>> keyboards.
>>>>>> True, and if the userspace is doing anything complicated here, the
>>>>>> hypervisor has to be responsible for ensuring that whatever it's doing
>>>>>> works with that, or else this system won't work.  I don't know that the
>>>>>> kernel is in a good position to police that.
>>>>> What do you mean by policing here?
>>>>>
>>>>> Isn't it the hypervisor that determines what virtual input devices will
>>>>> be available to the guest OS? Therefore, the hypervisor is in a
>>>>> position to expose exactly one pointer device and exactly one
>>>>> cursor plane to guest OS which means the guest OS cannot get the
>>>>> association wrong. If that's the general and expected hypervisor
>>>>> policy, then there is no need to design explicit device association in
>>>>> the guest kernel UAPI. If so, I'd like it to be mentioned in the kernel
>>>>> docs, too.
>>>> I'm not entirely sure how to fit what you're calling a "pointer" into my
>>>> mental model of what the hypervisor is doing...
>>> My definition: A pointer is a pointing input device that requires a
>>> cursor image to be drawn at the right location for it to be usable.
>> Right, but normal desktops (and our consoles) expect multiple input
>> devices to feed into a single cursor.  So the connection between the
>> on-screen cursor and the corresponding input-devices is not clear to me
>> when you start talking about multiple pointers, even without any
>> hypervisors in the picture.
> The connection is simple: there is an independent on-screen cursor for
> each logical pointer. How that cursor is drawn is irrelevant to the end
> user, and Wayland compositors (a type of a display server) will use any
> means necessary to draw it.
>
> Each logical pointer has one cursor that is independent from all other
> logical pointers. Each logical pointer can have any number of input
> devices controlling it. The assignments are decided by the userspace
> and implemented in a display (window system) server.
>
> This has been ingrained into the fundamental design of Wayland, even if
> the feature is rarely used in practise. The window system may expose
> multiple independent pointers to applications, and each pointer may also
> interact with the same window simultaneously. This naturally leads to
> the question "which cursor goes with which input device?", and the
> kernel, or anything below it, does not know that if there are multiple
> possibilities.

Right, but the whole notion of a "pointer" is also not a kernel concept 
as far as I know?  The kernel only knows that it's requested to display 
a particular set of screen contents on an particular CRTC output.  What 
we're asking for here is for the new HOTSPOT property to allow the final 
desired screen contents to have some opt-in flexibility on how it gets 
displayed to meet the needs of the para-virtualized drivers and their 
use of cursors.

I think trying to extend that beyond the display side to include the 
mouse/input semantics when those are not already part of the kernel 
interface isn't going to work well.

>
>
>>>> So I guess I'm not clear on what kind of usermode<=>kernel contract you
>>>> want here if the kernel isn't what's owning the translation between the
>>>> mouse input and the cursor position.  The hypervisor awkwardly has to
>>>> straddle both the input/graphics domain, and we do so by making
>>>> assumptions about how the user desktop is going to behave.
>>> Correct. In order to reduce that awkwardness, I encourage you to write
>>> down the expectations and requirements in this new Linux UAPI (the KMS
>>> cursor place hotspot property). Then you can be much more confident on
>>> how a random Linux desktop will behave.
>>>
>>> It will also help the reviewers here to understand what the new UAPI is
>>> and how it is supposed to work.
>> The cursor hotspot is I think fairly straightforward, as far as what
>> that means for how hypervisor clients are expected to draw the mouse,
>> and Zack's working on that part.
>>
>> My point was that how the hypervisor then sends input is sort of outside
>> the scope of the graphics portion here, and I think probably outside the
>> current Linux UAPI entirely (unless there's some other input/topology
>> system somewhere else I'm not familiar with).
> I would not say that the hotspot property is in any way obvious. I've
> spent my whole Wayland and compositor developer career of over 10 years
> never seeing the need of the kernel knowing the hotspot myself, because
> I never use VMWare like tools.
Sorry, I meant straight-forward as in well-defined, not as in obvious.  
It should definitely be documented.

We can just clearly specify the graphics contract here in a way that is 
trickier for the mouse/input parts.

>
> You cannot describe why hotspot property is needed or how it works
> without explaining the input side. The hotspot property is actually
> really weird, because its existence requires combining the input system
> with the graphics system for it to make sense. The input system used to
> be out of scope indeed, but this addition forces it in scope. This is
> the first time that I know of when the kernel or components below the
> kernel needs to know, hence there is no existing infrastructure in
> Linux to record that topology or anything.

I think the problem here is going to be that each of the userspaces and 
para-virtual drivers handle this differently...

They all have their own mechanisms to pass the topology/input-mapping up 
to the hypervisor (either custom IOCTLs on their graphics driver, or 
else backdoor-style from userspace), and how that works at the userspace 
layer (ie X11/Wayland) is differently.

>
> (Sidetrack: for ultra-fast displays, say 1000 Hz refresh for seamless
> hand-eye coordination, it could make very much sense for userspace to
> off-load arbitrary KMS plane positioning through the kernel into
> hardware in general, tying the plane position to some input device
> position temporarily. It might make some sense even with today's
> consumer hardware. So I don't think the concept is fundamentally
> limited to paravirtualized environments. But going there would need a
> lot more work than I am willing to suggest to VMWare to tackle to just
> make their own products better.)

Sure, but to do that we would need to standardize how the input mapping 
happens, and create standard kernel interfaces for the topology 
information, and relative mouse acceleration curves, etc. That's the 
piece that I'm saying isn't standardized enough now to be able to spell 
that out in documentation now, because there is no standard way of 
handling all that.

As a hand-waiving, "this is kind of how it works", it's useful to 
document.  But if you're pushing for a hard "this is how the Linux 
kernel handles mouse input", then I don't think any of these components 
are ready to commit to the mouse/input handling as a Linux UAPI.

>> So I guess I would vote for trying to include something to that effect
>> as context or explanation, but not try to strictly define how that works?
> Yes, exactly.

Okay, if we can keep the mouse/input stuff on the fuzzy side then I 
think we're on the same page.

Thanks!
--Michael Banack