Input and games.

[Todd Showalter]

On Fri, Apr 19, 2013 at 1:52 PM, Jonas Kulla <nyocurio at gmail.com> wrote:

> I can totally see where you're coming from (having worked on a small
> engine myself in the past), but I feel like creating static input states
> should always be done on client side. Especially in Wayland where
> "frame-perfectness" is crucial, round-trips such as input state polling
> are strongly discouraged.
> On the other hand, this doesn't mean that every developer has to
> reinvent the wheel. Input state caching could certainly be split off
> into a client library.

    That's perfectly workable; as I said elsewhere, we already do
essentially that on all other desktop environments, so it's not a
hardship.

> Well, at least with the "relative motion" proposal Pekka linked,
> such a grab would always be breakable by things such as AltTab.
> AFAIK the general consensus is that Wayland clients should never ever
> be able to "hold the display server hostage" as would often happen in X11
> with faulty clients (ie. unable to leave a fullscreen hanging game).
> So you shouldn't worry about that.

    Ok, that's what I was hoping.

> I think the biggest reason for "relative motion" is to prevent having to
> warp
> the pointer, which wayland won't support. Everything else, ie. how your
> client interprets keyboard/scroll events, is up to you. I don't think it's
> wise to let compositors send "text" events, that's the reason the keymap is
> provided as a fd. And it's certainly not a good idea to tell them to
> suddenly
> provide different events for the same physical buttons/keys.

    From a game point of view, when we want relative motion we often
don't want to see the pointer at all; the Quake example being an
example.  On the other hand, sometimes we want to park the pointer
somewhere (like the edge of a window) and use pointer motion as a
delta to apply to an object (like scrolling a map).

>>     There's also the question of clean recovery; if a game has changed
>> the video mode (if that's allowed any more, though these days with LCD
>> panels and robust 3D hardware maybe that's just a bad idea), turned
>> off key repeat and captured the mouse, all of that needs to be
>> reverted if the game exits ungracefully.  Which sometimes happens,
>> especially during development.
>
> True. But as I mentioned above, what is even more critical than that is the
> ability to escape _hanged_ clients without having to reboot your PC.
> I think in that area the wayland devs are already cautious about doing the
> right thing. Global (compositor) hotkeys/shortcuts will never be able to
> be swallowed by clients AFAIK.

    By "exits ungracefully" I mean "hanged", "died with sigsegv",
"died with sigbus", "deadlocked" or what have you.  Anything where the
game can no longer be relied upon to undo the state changes it
requested.

> Ok, now this seems REALLY confusing to me. In over 10 years of using
> computers I have never come across a single application behaving like that.
> Could you maybe name an example? Also, isn't this the reason mouse
> wheels were invented in the first place? I can see where you're coming
> from, but for me personally scroll bars have always been more of an
> "absolute" control widget, ie. I use them when I want to go exactly to
> the beginning/end/40% mark of a viewport.

    I'm stuck on a Mac at the moment, unfortunately, so I can't give
you verified Linux examples, but I'm writing this email in the GMail
web client on Firefox.  If I grab the scroll thumb and drag it
upwards, the mouse pointer moves far more slowly than if I just moved
the mouse up without grabbing the thumb, and I see all of the text
scroll by.  If the thumb was moving at the speed my mouse normally
moves, I'd have jumped over 80% of the text.  LibreOffice is showing
me the same behavior, and I remember similar behavior lots of other
places.  IIRC it was also pretty much standard behavior on
well-behaved Win32 programs, at least from Win95 or so until XP; I've
managed to largely avoid Windows since then, so I couldn't tell you
what it does these days.

    The thing is, with this model the thumb *position* is still
absolute; if you scroll the thumb to 40% of the way down the bar, the
view shows you 40% of the way down the document.  The difference is
that instead of warping the document past the view at the thumb's rate
of travel, the thumb's rate of travel is accelerated or attenuated to
match a sane document scrolling speed, and the pointer is warped to
keep it over the thumb.  It's a subtle effect, but I bet you'll find
that a lot of the software you use actually does it.  I've definitely
seen this built to respect pointer acceleration as well, so that if
you really do yank the mouse, the document just flies by.  But if
you're moving at a normal rate, the pointer moves slowly enough that
the document scrolls at a reasonable speed.

    The important thing is, without that ability, you simply can't
scroll long documents sanely with a scroll bar thumb.  I suppose you
could fudge it by grabbing and ungrabbing the pointer during
scrolling, but it would definitely be a fudge.  Without the pointer
deceleration on thumb drag, you simply can't do fine positioning with
the scroll thumb once the ratio of the obscured (ie: offscreen)
document length to the thumb travel length becomes too great.

    I suppose you could also handle this with pointer sensitivity
modification; if you know the document:view ratio is 10:1, you could
scale the pointer speed by 0.1 when the scroll wheel was grabbed while
not allowing warping.

> Also, I'm pretty sure something like mouse warping will likely never be
> implemented in wayland for design reasons (you don't want malicious
> apps controlling crucial variables such as pointer location).

    How does that differ from input grab, though?  Surely the thing to
do is say that you can only warp the pointer if you have focus, and
your model already deals with breaking grabs on focus loss.

> Again, I don't think any of the major toolkits (Qt, GTK+, EFL..)
> implement this, and I sure haven't seen it on Windows/Mac before,
> so where is this concept originating from?

    It's a subtle effect and easy to miss, but I think you'll find it
it's there in software you use.  It's happening on this mac I'm using
right now, and it at least used to happen in windows.

> That I think is a valid use case. I'm not sure how it would be implemented
> (maybe with a simple "pointer barrier on this surface" request that
> can be broken with AltTab and Co?).

    That would work fine.

> As already mentioned, Wayland has made sure applications can't block
> compositor level shortcuts and the like, so we're good here.

    Perfect.

> As Pekka said, input events are dispatched once per frame, so ideally
> you'd never get events more than a frame old, in which case the timestamp
> might provide a small hint to the simulation. But in case of (online)
> multiplayer you'd probably not constrain the simulation clock to a single
> client's monitor refresh rate, so I think you have a good point that this
> can be a downfall of such event delivery.

    If it's once per frame, that's good enough.  The latency inherent
in online means that 60Hz quantization is the least of our worries.

> As Pekka said, only the base is arbitrary, the value is defined as
> milliseconds, so as you described you could retain one value at
> simulation start and calculate deltas from thereon. The only problem
> is that the timestamp will overflow and wrap around at some point,
> so you would have to somehow catch that (maybe update the
> the delta base at regular intervals). On the other hand, for that to
> happen it'll probably take days, if not weeks.

    That works fine.

                                             Todd.

--
 Todd Showalter, President,
 Electron Jump Games, Inc.