Compositor grabs (was: Re: [PATCH] protocol: Add DnD actions)

[Jonas Ådahl]

On Tue, Jun 09, 2015 at 07:26:15PM +0200, Carlos Garnacho wrote:
> Hey Jonas :),
> 
> On jue, 2015-06-04 at 12:34 +0800, Jonas Ådahl wrote:
> > On Fri, Apr 17, 2015 at 02:16:41PM +0200, Carlos Garnacho wrote:

<snip>

> > > 
> > > Let's expand on that example, maybe far-streched, but certainly 
> > > possible:
> > > - I'm manipulating a client window with 2 fingers on the 
> > > touchscreen
> > > (say zooming an image)
> > > - Any other interaction on the client makes it pop up an xdg_popup
> > > (say a third touch, a key combo, or the pointer)
> > > - Q: what happens with the two first touches?
> > > 
> > > Ideally, these touches should be "cancelled" on the first surface
> > > (wl_touch.cancelled seems to be per client, not per-surface 
> > > though),
> > > and stay non-reactive within the grab, so they wouldn't trigger
> > > anything unintended (they're implicitly grabbed on another surface
> > > after all)
> > 
> > It seems rather unspecified in the protocol, but AFAIK at least in
> > weston only one surface may have any touch points active at any given
> > time. Mutter seems to do the opposite, i.e. any number of surfaces 
> > can
> > have "touch focus". 
> 
> TBH I did this on purpose on mutter, again the protocol is quite open
> in this regard, so I made each touch go where its implicit grab
> defines. This doesn't go against anything in the protocol (eg. touch
> "focus" doesn't imply keyboard focus), and it seemed to me like
> artificially capping what we're able to.

Considering that with the current version it is impossible to cancel
touches on an individual surface, i.e. we can't implement the semantics
discussed, should we maybe only support single surface touch points
and then add support for it when we can cancel touches on a per surface
basis?

> 
> > I'm don't know what is intentional here, but
> > assuming single touch focus is, wl_touch.cancelled being per client 
> > is
> > not a problem when an explicit grab is initiated on another surface,
> > since we'd have to cancel the touches of the previously focused 
> > surface
> > (which is all on that client), and then start them on the new one 
> 
> 
> You've got a point here, although the questions still hold for
> pointer/keyboard.
> 
> > (hmm,
> > should the popup get new 'down' events? I assume it must, since one
> > should not have to raise the finger and then touch again to interact
> > with the popup).
> 
> Hmm, I'm not sure we should, it IMO introduces inconsistency for this
> one usecase. I would much prefer to encourage wl_subsurface if such
> thing is a must, this is grabless on the compositor, so the client can
> define its own semantics. 

Yea, mabe it makes more sense to not support it and not provide
xdg_popup like semantics for that use case.

> 
> I eg. wonder what would happen with coordinates laid outside the
> surface, do we get "down" events anyway? Or, say a popup containing a
> scrolled view is shown so its triggering touch lies in the middle, do
> we want it to scroll kinetic scrolling, select an item or whatnot?
> 
> Pointers aren't that much of an issue because press/released is
> separated semantically from enter/leave, on wl_touch these are put
> together into down/up. IMO, fixing this requires either separating
> these likewise here, or refining wl_touch.cancelled somehow.

The difference is that, for a popup that was clicked but not released,
the new surface will still receive motion events, and can by itself keep
track of whether the button was clicked or not. For touches, if we don't
receive a 'down', and the previous touches are cancelled, there won't be
any new motion events. The biggest difference would be that for a
xdg_popup like menu, with a pointer we could
click-hold-drag-select-release, but for touch, for the same menu, we
would require touch-release-touch-release for the same action.

> 
> > 
> > Or, the other way, where there is no such thing as single touch 
> > focus.
> > I.e. that multiple surfaces can have touch points at the same time, 
> > then
> > we'd need changing the protocol to support what you describe (as as 
> > you
> > say, wl_touch.cancelled is per client).
> > 
> > Anyway, either we need to properly clarify that in wl_touch I think, 
> > and
> > it'd be helpful to know if this was considered when designing the
> > protocol.
> 
> Would be helpful indeed :).

Given how wl_touch.cancelled works, I would at least assume its
single-surface, but that, as I said, just my assumption.

> 
> > 
> > > 
> > > Currently, on the weston code, focusing a bit on the all-touch 
> > > case,
> > > it actually happens the worst that could happen, the xdg_popup 
> > > touch
> > > grab redirects already started touch sequences to the grabbing 
> > > surface
> > > right away, and the original surface will be deaf to them of a 
> > > sudden,
> > > leading to inconsistent state on both the original and the grabbing
> > > surface wrt those touch sequences. The DnD touch grab doesn't fare
> > > much better, it will ignore other touches than the one starting the
> > > drag, so the pre-grab touches would effectively go nowhere, and 
> > > AFAICS
> > > similar issues arise with pointer grabs vs touch.
> > 
> > Yes this seems wrong.
> > 
> > > 
> > > With keyboards, it happens likewise, if compositors are to possibly
> > > consume events there, focus should move out of the previous 
> > > surface.
> > > IMO, any grabbing model that does not redirect all input, nor 
> > > ensures
> > > a coherent state in clients is calling for trouble...
> > > 
> > > In the X11 world, this would roughly be a Virtual Core
> > > Pointer+Keyboard grab (not that touch and active grabs are trouble
> > > free in X11, but...), GTK+ for example does grab both devices on 
> > > every
> > > of those grabbing places wayland/xdg protocols are trying to cater 
> > > for
> > > (I've even pondered about adding a gdk_device_grab_pair() for 
> > > years).
> > > 
> > > I think some consistent model should be devised here, and embedded
> > > into the protocol (docs).
> > 
> > I agree that some consistency would be good. How do you suggest we do 
> > so?Should we make it globally enforced semantics, meaning formally
> > specifying what an explicit grab is, or should it be per protocol 
> > that
> > uses them?
> 
> I started doing a patch with a model I found sound, the approach ended
> up being mixed, with each input interface defining its expected
> behavior, and adding specific notes on the places grabs take place.
> 
> 
> >  I wonder if there will be explicit grab types that require
> > special casing. For example the discussed pointer lock/confine takes
> > something similar to an explicit grab, and it seems reasonable to 
> > make
> > this grab also take the keyboard focus, should it also take the touch
> > focus?
> 
> Yeah I think so, at least with a "touches elsewhere will be ignored"
> explicit note. Hmm, although it'd look a bit awkward if pointer
> confinement is part of wl_pointer, I'll double check these protocol
> patches.
> 
> > 
> > > 
> > > > 
> > > > > - Ensuring the grab affects the routing of all devices/events, 
> > > > > and
> > > > > that no client gets partial streams
> > > > 
> > > > You mean that pointer grab should implicitly grab the touch and 
> > > > keyboard
> > > > device as well? What do you mean with partial stream?
> > > 
> > > See the example above with surfaces receiving not properly
> > > started/finalized series of event for touch sequences, or the
> > > suggestions in the DnD thread about having drag source/dest handle
> > > keyboard events, while the compositor is certainly interested in
> > > handling/consuming some keyboard events too.
> > 
> > Well, the compositor personally doesn't care about grabs. It can do
> > whatever it wants. But if we want the grab consistency you are 
> > talking
> > about, do you mean that the source should loose keyboard focus when 
> > the
> > grab is initiated? 
> 
> Yes I think so, the compositor can sure intercept modifiers for actions
> and forward anything else to the drag source, but this looks
> inconsistent if the client doesn't know any longer which keycodes can
> it receive and which it can't.

Isn't this the case already? The compositor may eat arbitrary keys given
it has internal key bindings that does so.

> 
> That's a reason why, in the case we did source/dest action decision in
> DnD, I would much prefer to make a wl_data_source/offer.modifiers than
> mixing wl_data_source and wl_keyboard together.

I agree with this.


Jonas

> 
> > Only for Xwayland surfaces the pointer focus will be
> > left intact, regular ones will loose it.
> 
> XDND/Xwayland is going to look otherworldly anyway. I guess it also
> depends on the type of "grab", pointer confinement would be pretty much
> as-is AFAICS, if we ever relay confine_to from XGrabPointer requests.
> 
> So, I'm sending patches for wayland/xdg-shell protocols, consider this
> a RFC so there's no weston implementation of the details yet.
> 
> Cheers,
>   Carlos