[PATCH v4 2/8] drm/atomic: Add support for mouse hotspots

[Michael Banack]

On 7/7/23 01:38, Pekka Paalanen wrote:
>
> That statement was based on the assumption that existing hypervisors
> and VM viewer applications are not prepared to deal with hotspots
> outside of cursor image. Therefore, if a guest is upgraded to a version
> that uses hotspots outside of cursor images, and the VM stack is not
> updated, it will malfunction.
>
> Therefore it is best to model the new UAPI in a way that is compatible
> with existing VM stack, especially since allowing this new feature
> (hotspots outside of cursor image) has no known benefits.
>
> Below I see my assumption was incorrect, but it still causes you to
> fall back to something less optimal.

Okay, right.  That's why I was saying that it's not a big deal either 
way to VMware, but I wanted to at least make the case for allowing 
somewhat arbitrary hotspots just because it is semantically meaningful, 
and I don't know if any other hypervisors care about allowing it more 
than we do.


>> Essentially setting the hotspot properties means that the hypervisor
>> console can choose to either draw the cursor where the plane is located,
>> or use the cursor-plane + hotspot information to draw the cursor where
>> the user's mouse is on the client.
>>
>> That works the same whether the hotspot is clamped or not.  We mostly
>> use clamping to avoid pathological cases (like a hotspot ot MAX_UINT32),
>> and get away with it because real Guest applications that do this are
>> very rare.
> My point here is that you can design the new Linux UAPI to help you:
> you can rule out cases that would lead to non-optimal behaviour, like
> falling back to the drawing of cursor plane you mention when it would
> be better to commandeer the cursor plane with the hotspot information.

We can't though, because we can't trust the guest kernel any more than 
the kernel can trust userspace.

So we need to handle these cases one way or another, both for older 
guests, and to ensure we don't have some security issue from a malicious 
guest kernel.

>
>>>>>>> The question of which input device corresponds to which cursor plane
>>>>>>> might be good to answer too. I presume the VM runner is configured to
>>>>>>> expose exactly one of each, so there can be only one association?
>>>>>> As far as I know, all of the VM consoles are written as though they
>>>>>> taking the place of what would the the physical monitors and input
>>>>>> devices on a native machine.  So they assume that there is one user,
>>>>>> sitting in front of one console, and all monitors/input devices are
>>>>>> being used by that user.
>>>>> Ok, but having a single user does not mean that there cannot be
>>>>> multiple independent pointers, especially on Wayland. The same with
>>>>> keyboards.
>>>> True, and if the userspace is doing anything complicated here, the
>>>> hypervisor has to be responsible for ensuring that whatever it's doing
>>>> works with that, or else this system won't work.  I don't know that the
>>>> kernel is in a good position to police that.
>>> What do you mean by policing here?
>>>
>>> Isn't it the hypervisor that determines what virtual input devices will
>>> be available to the guest OS? Therefore, the hypervisor is in a
>>> position to expose exactly one pointer device and exactly one
>>> cursor plane to guest OS which means the guest OS cannot get the
>>> association wrong. If that's the general and expected hypervisor
>>> policy, then there is no need to design explicit device association in
>>> the guest kernel UAPI. If so, I'd like it to be mentioned in the kernel
>>> docs, too.
>> I'm not entirely sure how to fit what you're calling a "pointer" into my
>> mental model of what the hypervisor is doing...
> My definition: A pointer is a pointing input device that requires a
> cursor image to be drawn at the right location for it to be usable.
Right, but normal desktops (and our consoles) expect multiple input 
devices to feed into a single cursor.  So the connection between the 
on-screen cursor and the corresponding input-devices is not clear to me 
when you start talking about multiple pointers, even without any 
hypervisors in the picture.

>
>> For a typical Linux Guest, we currently expose 3+ virtual mouse devices,
>> and choose which one to send input to based on what their guest drivers
>> are doing, and what kind of input we got from the client.  We expect the
>> input from all of those to end up in the same user desktop session,
>> which we expect to own all the virtual screens, and that the user the
>> only gets one cursor image from that.
> Why do you need to expose so many pointer devices? Just curious.
For one, we don't know what drivers are actually going to be running in 
the Guest.  If someone configured the Guest to not support the PS/2 
mouse, or didn't have USB support compiled in, we still need to be able 
to send mouse input.  (We actually ran into this for years with some 
Linux distro installers, where they had frozen their installer with some 
weird/older kernel configs and just didn't support our preferred vmmouse 
device.)  So we plug them all in at boot, and then try to pick the one 
that looks the most active.

But we also need to be able to send both absolute/relative events, and 
we had trouble getting Guests to support both of those coming from the 
same virtual mouse device, so if the client changes mouse modes we would 
route those to the appropriate device dynamically.

There's some other quirky situations, like some absolute virtual mice 
not supporting the entire multimon topology correctly or mouse buttons 
not applying properly when things get split across mouse devices, but 
those are less common.

>
> If you do expose multiple mouse (pointer) devices, then guest OS can
> choose to use each of them as a different independent cursor on the
> same desktop. The only thing stopping that is that it's not usually
> useful, so it's not done.
>
> Therefore, what you need to document in the Linux UAPI instead is that
> *all* pointer devices are associated with the *same* cursor plane. That
> forbids the multi-pointer pointer scenario the VM stack cannot handle.
At least all mouse input devices that the hypervisor console is going to 
use to send input to that desktop, yeah.  (You could still have 
non-hypervisor input sources that don't enter the picture, like some 
kind of passthrough/remote device or something.)

>> So I guess I'm not clear on what kind of usermode<=>kernel contract you
>> want here if the kernel isn't what's owning the translation between the
>> mouse input and the cursor position.  The hypervisor awkwardly has to
>> straddle both the input/graphics domain, and we do so by making
>> assumptions about how the user desktop is going to behave.
> Correct. In order to reduce that awkwardness, I encourage you to write
> down the expectations and requirements in this new Linux UAPI (the KMS
> cursor place hotspot property). Then you can be much more confident on
> how a random Linux desktop will behave.
>
> It will also help the reviewers here to understand what the new UAPI is
> and how it is supposed to work.

The cursor hotspot is I think fairly straightforward, as far as what 
that means for how hypervisor clients are expected to draw the mouse, 
and Zack's working on that part.

My point was that how the hypervisor then sends input is sort of outside 
the scope of the graphics portion here, and I think probably outside the 
current Linux UAPI entirely (unless there's some other input/topology 
system somewhere else I'm not familiar with).

> However, in your case, the userspace (Wayland or X11 display server) is
> not free to choose any arbitrary input-cursor association they might
> want. You have a specific expectation that all pointing devices control
> the same pointer. Since the hotspot property is exclusive to your use
> case, I think it make sense to write down the expectations with the
> hotspot property. Guest userspace has to explicitly program for the
> hotspot property anyway, so it can also take care of your requirements.
I see your point, and I can see the value in documenting something to 
that effect, if only because it's /useful/ for userspaces trying to work 
with this.  (And the only way anyone is using this today.)

But I'm still a little fuzzy on what portion of that is within the Linux 
UAPI scope for the hotspot...

It seems like it might be more useful to restrict the scope of the Linux 
UAPI contract for how the hotspot property works to just how userspace 
can expect the hypervisors to display it on screen, and not try to tie 
in any expectations for how mouse input is going to work.

Like, VMware is using virtual mouse devices here, but another hypervisor 
might have no kernel mouse device and instead inject input entirely 
through a userspace daemon?  So even trying to express the input part of 
the contract in terms of kernel input devices I'm not sure makes sense.

I guess my fear is that I don't want to lock this down in a way that 
excludes some well-meaning hypervisor that implements the graphics part 
correctly but does something just weird enough on the input side to not 
be considered compliant.

So I guess I would vote for trying to include something to that effect 
as context or explanation, but not try to strictly define how that works?

--Michael Banack