Compositor grabs (was: Re: [PATCH] protocol: Add DnD actions)

[Carlos Garnacho]

Hey Jonas :),

On jue, 2015-06-04 at 12:34 +0800, Jonas Ådahl wrote:
> On Fri, Apr 17, 2015 at 02:16:41PM +0200, Carlos Garnacho wrote:
> > Hey Jonas,
> > 
> > This is drifting a bit off the topic of the original thread, better 
> > to
> > spin this off. I'll reply to the DnD bits in another email.
> 
> Hey, sorry for the delay. Some comments/questions below.
> 
> > 
> > On Fri, Apr 17, 2015 at 9:50 AM, Jonas Ådahl <jadahl at gmail.com> 
> > wrote:
> > > On Thu, Apr 16, 2015 at 12:55:31PM +0200, Carlos Garnacho wrote:
> > > > Hey Jonas,
> > > > 
> > > > On Thu, Apr 16, 2015 at 10:15 AM, Jonas Ådahl <jadahl at gmail.com
> > > > > wrote:
> > > > 
> > > > <snip>
> > > > 
> > > > > 
> > > > > I'd have to agree on that it doesn't seem like the best thing 
> > > > > to let the
> > > > > compositor choose the preferred action. Having it apply 
> > > > > compositor
> > > > > specific policy given what the keyboard state or similar will 
> > > > > probably
> > > > > never work out very well, given that for example what 
> > > > > modifier state
> > > > > means what type of action is very application dependent.
> > > > > 
> > > > > On the other hand, I'm not sure we can currently rely on 
> > > > > either side
> > > > > having keyboard focus during the drag. In weston the source 
> > > > > will have the
> > > > > focus because starting the drag was done with a click which 
> > > > > gave the
> > > > > surface keyboard focus implicitly, but what'd happen if the 
> > > > > compositor
> > > > > has keyboard-focus-follows-mouse? We could probably say that 
> > > > > drag implies
> > > > > an implicit grab on another device on the same seat to 
> > > > > enforce no
> > > > > changing of keyboard focus, but not sure that is better.
> > > > 
> > > > In gtk+/gnome we currently have the following keybindings 
> > > > active during DnD:
> > > > 
> > > > - Cursor keys move the drag point, modifiers affect speed
> > > > - Esc key cancels drag
> > > > - Modifiers alone pick an action from the offered list
> > > > 
> > > > So ok, the latter is dubious to punt to compositors, but 
> > > > there's
> > > > basically no other choice with the 2 first ones.
> > > > 
> > > > More generally, I have the opinion that compositors grabs 
> > > > should
> > > > behave all consistently, as in:
> > > > 
> > > > - Ensuring clients reset all input state (we eg. don't cancel 
> > > > ongoing
> > > > touches when xdg_popup/dnd/... grabs kick in)
> > > 
> > > What does "client reset all input state" mean? What state can a 
> > > client
> > > reset?
> > 
> > Let's expand on that example, maybe far-streched, but certainly 
> > possible:
> > - I'm manipulating a client window with 2 fingers on the 
> > touchscreen
> > (say zooming an image)
> > - Any other interaction on the client makes it pop up an xdg_popup
> > (say a third touch, a key combo, or the pointer)
> > - Q: what happens with the two first touches?
> > 
> > Ideally, these touches should be "cancelled" on the first surface
> > (wl_touch.cancelled seems to be per client, not per-surface 
> > though),
> > and stay non-reactive within the grab, so they wouldn't trigger
> > anything unintended (they're implicitly grabbed on another surface
> > after all)
> 
> It seems rather unspecified in the protocol, but AFAIK at least in
> weston only one surface may have any touch points active at any given
> time. Mutter seems to do the opposite, i.e. any number of surfaces 
> can
> have "touch focus". 

TBH I did this on purpose on mutter, again the protocol is quite open
in this regard, so I made each touch go where its implicit grab
defines. This doesn't go against anything in the protocol (eg. touch
"focus" doesn't imply keyboard focus), and it seemed to me like
artificially capping what we're able to.

> I'm don't know what is intentional here, but
> assuming single touch focus is, wl_touch.cancelled being per client 
> is
> not a problem when an explicit grab is initiated on another surface,
> since we'd have to cancel the touches of the previously focused 
> surface
> (which is all on that client), and then start them on the new one 


You've got a point here, although the questions still hold for
pointer/keyboard.

> (hmm,
> should the popup get new 'down' events? I assume it must, since one
> should not have to raise the finger and then touch again to interact
> with the popup).

Hmm, I'm not sure we should, it IMO introduces inconsistency for this
one usecase. I would much prefer to encourage wl_subsurface if such
thing is a must, this is grabless on the compositor, so the client can
define its own semantics. 

I eg. wonder what would happen with coordinates laid outside the
surface, do we get "down" events anyway? Or, say a popup containing a
scrolled view is shown so its triggering touch lies in the middle, do
we want it to scroll kinetic scrolling, select an item or whatnot?

Pointers aren't that much of an issue because press/released is
separated semantically from enter/leave, on wl_touch these are put
together into down/up. IMO, fixing this requires either separating
these likewise here, or refining wl_touch.cancelled somehow.

> 
> Or, the other way, where there is no such thing as single touch 
> focus.
> I.e. that multiple surfaces can have touch points at the same time, 
> then
> we'd need changing the protocol to support what you describe (as as 
> you
> say, wl_touch.cancelled is per client).
> 
> Anyway, either we need to properly clarify that in wl_touch I think, 
> and
> it'd be helpful to know if this was considered when designing the
> protocol.

Would be helpful indeed :).

> 
> > 
> > Currently, on the weston code, focusing a bit on the all-touch 
> > case,
> > it actually happens the worst that could happen, the xdg_popup 
> > touch
> > grab redirects already started touch sequences to the grabbing 
> > surface
> > right away, and the original surface will be deaf to them of a 
> > sudden,
> > leading to inconsistent state on both the original and the grabbing
> > surface wrt those touch sequences. The DnD touch grab doesn't fare
> > much better, it will ignore other touches than the one starting the
> > drag, so the pre-grab touches would effectively go nowhere, and 
> > AFAICS
> > similar issues arise with pointer grabs vs touch.
> 
> Yes this seems wrong.
> 
> > 
> > With keyboards, it happens likewise, if compositors are to possibly
> > consume events there, focus should move out of the previous 
> > surface.
> > IMO, any grabbing model that does not redirect all input, nor 
> > ensures
> > a coherent state in clients is calling for trouble...
> > 
> > In the X11 world, this would roughly be a Virtual Core
> > Pointer+Keyboard grab (not that touch and active grabs are trouble
> > free in X11, but...), GTK+ for example does grab both devices on 
> > every
> > of those grabbing places wayland/xdg protocols are trying to cater 
> > for
> > (I've even pondered about adding a gdk_device_grab_pair() for 
> > years).
> > 
> > I think some consistent model should be devised here, and embedded
> > into the protocol (docs).
> 
> I agree that some consistency would be good. How do you suggest we do 
> so?Should we make it globally enforced semantics, meaning formally
> specifying what an explicit grab is, or should it be per protocol 
> that
> uses them?

I started doing a patch with a model I found sound, the approach ended
up being mixed, with each input interface defining its expected
behavior, and adding specific notes on the places grabs take place.


>  I wonder if there will be explicit grab types that require
> special casing. For example the discussed pointer lock/confine takes
> something similar to an explicit grab, and it seems reasonable to 
> make
> this grab also take the keyboard focus, should it also take the touch
> focus?

Yeah I think so, at least with a "touches elsewhere will be ignored"
explicit note. Hmm, although it'd look a bit awkward if pointer
confinement is part of wl_pointer, I'll double check these protocol
patches.

> 
> > 
> > > 
> > > > - Ensuring the grab affects the routing of all devices/events, 
> > > > and
> > > > that no client gets partial streams
> > > 
> > > You mean that pointer grab should implicitly grab the touch and 
> > > keyboard
> > > device as well? What do you mean with partial stream?
> > 
> > See the example above with surfaces receiving not properly
> > started/finalized series of event for touch sequences, or the
> > suggestions in the DnD thread about having drag source/dest handle
> > keyboard events, while the compositor is certainly interested in
> > handling/consuming some keyboard events too.
> 
> Well, the compositor personally doesn't care about grabs. It can do
> whatever it wants. But if we want the grab consistency you are 
> talking
> about, do you mean that the source should loose keyboard focus when 
> the
> grab is initiated? 

Yes I think so, the compositor can sure intercept modifiers for actions
and forward anything else to the drag source, but this looks
inconsistent if the client doesn't know any longer which keycodes can
it receive and which it can't.

That's a reason why, in the case we did source/dest action decision in
DnD, I would much prefer to make a wl_data_source/offer.modifiers than
mixing wl_data_source and wl_keyboard together.

> Only for Xwayland surfaces the pointer focus will be
> left intact, regular ones will loose it.

XDND/Xwayland is going to look otherworldly anyway. I guess it also
depends on the type of "grab", pointer confinement would be pretty much
as-is AFAICS, if we ever relay confine_to from XGrabPointer requests.

So, I'm sending patches for wayland/xdg-shell protocols, consider this
a RFC so there's no weston implementation of the details yet.

Cheers,
  Carlos