Input and games.

[Jonas Kulla]

2013/4/19 Todd Showalter <todd at electronjump.com>

> On Fri, Apr 19, 2013 at 5:18 AM, Pekka Paalanen <ppaalanen at gmail.com>
> wrote:
>

> > Event driven is a little more work for the "simple" games, but it gives
> > you guarantees. Would you not agree?
>
>     We can definitely work with it.  As much as anything it's a
> question of convenience; the question is really how much
> superstructure we need to build on top to get what we need.  We've
> already got that superstructure elsewhere, so porting it over is
> simple enough.  It would be more convenient if we didn't have to, but
> it's not a deal breaker.
>
>     For context, I'm not trying to convince you to change the protocol
> or the model per se; aside from anything else, I don't yet understand
> it well enough to seriously critique it.  A large part of what I'm
> hoping to do here is offer some insight into how games tend to use
> input, the kind of needs games often have, and the sorts of
> considerations that make a system easier or harder to put a game on.
> Wayland obviously has competing considerations, some of which are
> arguably more important than games.  If one can imagine such a thing.
>
>     One thing worth noting here is why we want operate on virtualized
> input structures rather than raw events.  One reason I mentioned
> above; accumulating events so that they can be applied between frames.
>  Another reason is complexity management; games can be quite complex
> beasts consisting of many parts, and everything that can be done to
> isolate those parts makes the game easier to develop and maintain.
>
>     The classic problem with a purely event-driven program is that
> somewhere in it there is a giant event loop that knows about
> everything in the program.  In something simple like a calculator,
> it's not a problem, but once you scale up to a large system with
> multiple subsystems the event loop can turn into a nightmare.  Having
> virtualized input structures that the game can query means that input
> tests can be isolated to the code where they belong. ie:
>
> if(KeyTrigger(KEY_D) && KeyDown(KEY_CTRL))
> {
>   Log("heap integrity %d\n", check_heap_integrity());
> }
>
>     You can achieve some of the same modularity with function pointer
> lists or similar hooks, but having a virtualized input structure has
> (in my experience at least) been the cleanest abstraction.


I can totally see where you're coming from (having worked on a small
engine myself in the past), but I feel like creating static input states
should always be done on client side. Especially in Wayland where
"frame-perfectness" is crucial, round-trips such as input state polling
are strongly discouraged.
On the other hand, this doesn't mean that every developer has to
reinvent the wheel. Input state caching could certainly be split off
into a client library.


>
> > Is this referring to the problem of "oops, my mouse left the Quake
> > window when I tried to turn"? Or maybe more of "oops, the pointer hit
> > the monitor edge and I cannot turn any more?" I.e. absolute vs.
> > relative input events?
>
>     Partly.  The issue is that *sometimes* a game wants the mouse and
> keyboard to behave in the standard way (ie: the mouse controls the
> pointer and lets you click gui elements, the keyboard is for entering
> text and hitting control keys) and *sometimes* the game wants the
> mouse motion to control an in-game object (often the camera) and just
> wants the keyboard and mouse buttons to be a big bag of digital
> buttons.  With the Quake example, when the pause menu is up, or when
> the terminal has been called down, the game wants the keyboard to be
> generating text commands on the terminal and the mouse to be able to
> select text and click on buttons.  When the terminal is gone and the
> game isn't paused, Quake wants the mouse to control the camera view
> and the keyboard WASD keys are emulating a game controller dpad.
>
>     So, yes, absolute vs. relative events is part of the issue, but
> it's also part of a greater context; whether the keyboard is
> generating strings or digital inputs, whether the mouse is generating
> positions or deltas, under what circumstances focus is allowed to
> leave the window, whether the mouse pointer is visible, and things
> like how system-wide hotkeys factor in to things.  Can I capture the
> keyboard and mouse without preventing the user from using alt-tab to
> switch to another program, for instance?
>

Well, at least with the "relative motion" proposal Pekka linked,
such a grab would always be breakable by things such as AltTab.
AFAIK the general consensus is that Wayland clients should never ever
be able to "hold the display server hostage" as would often happen in X11
with faulty clients (ie. unable to leave a fullscreen hanging game).
So you shouldn't worry about that.


>
>     Clean, fast switching between these states is part of it as well;
> in a game like Quake, as above, you want to be able to capture the
> mouse when the game is playing, but "uncapture" it when the pause menu
> or the game terminal are up, or if the player switches focus to
> another program.  In an RTS, you might want a visible cursor but want
> to constrain the mouse to the window to allow the map to scroll.  You
> might want to use the keyboard mostly for hotkeys, but if they hit
> enter you want them to be able to type a string in to broadcast to
> multiplayer chat.  The scroll wheel might control either the message
> scrollback or the zoom level, depending on what the cursor is floating
> over.
>

I think the biggest reason for "relative motion" is to prevent having to
warp
the pointer, which wayland won't support. Everything else, ie. how your
client interprets keyboard/scroll events, is up to you. I don't think it's
wise to let compositors send "text" events, that's the reason the keymap is
provided as a fd. And it's certainly not a good idea to tell them to
suddenly
provide different events for the same physical buttons/keys.


>     There's also the question of clean recovery; if a game has changed
> the video mode (if that's allowed any more, though these days with LCD
> panels and robust 3D hardware maybe that's just a bad idea), turned
> off key repeat and captured the mouse, all of that needs to be
> reverted if the game exits ungracefully.  Which sometimes happens,
> especially during development.
>

True. But as I mentioned above, what is even more critical than that is the
ability to escape _hanged_ clients without having to reboot your PC.
I think in that area the wayland devs are already cautious about doing the
right thing. Global (compositor) hotkeys/shortcuts will never be able to
be swallowed by clients AFAIK.


>
> > There is a relative motion events proposal for mice:
> >
> http://lists.freedesktop.org/archives/wayland-devel/2013-February/007635.html
>
>     Something like that will be needed for a lot of styles of game,
> but also has use elsewhere.  For example, there used to be a widget on
> Irix machines IIRC, that looked like a trackball.  If you put the
> mouse pointer on it, held down the mouse button and then moved the
> mouse, it would scroll the trackball control rather than move the
> mouse pointer.
>
>     Similarly, when you're doing scroll bars, if you want to make a
> scroll bar where dragging the thumb moves the scrolled view at a rate
> that is pixel-proportional rather than window-size proportional, you
> have to be able to warp the pointer; otherwise, the view is slaved to
> the thumb position, so taller views scroll by faster.
>
>     Concrete example:  Let's say I have a document that is 1000 pixels
> tall, in a view that's 400 pixels tall.  Let's fudge the math a bit,
> say the thumb is one pixel tall and the region the thumb can be
> scrolled over is the full height of the window.  The window shows 40%
> of the document.  Without pointer warping, each step in the scroll bar
> is (600 / 400) pixels, so we're scrolling on average 1.5 pixels of
> view for every pixel the thumb moves up or down the screen.
>
>     Now, in the same view, we have a 250000 pixel tall document.  The
> document got longer, but the scroll bar is the same height (and thus,
> the same number of steps).  Each step of the scroll bar is now (249600
> / 400), or 624 pixels, enough that each scroll thumb movement scrolls
> more than 1.5x the view area.
>
>     The classic solution to this is when the scroll amount goes above
> or below sane thresholds, the view is moved by a sane amount, the
> scroll bar is moved by the correct amount (if any) for the new view,
> and if necessary the pointer is warped to the new thumb position.
>

Ok, now this seems REALLY confusing to me. In over 10 years of using
computers I have never come across a single application behaving like that.
Could you maybe name an example? Also, isn't this the reason mouse
wheels were invented in the first place? I can see where you're coming
from, but for me personally scroll bars have always been more of an
"absolute" control widget, ie. I use them when I want to go exactly to
the beginning/end/40% mark of a viewport.
Also, I'm pretty sure something like mouse warping will likely never be
implemented in wayland for design reasons (you don't want malicious
apps controlling crucial variables such as pointer location).



> > Clients cannot warp the pointer, so there is no way to hack around it.
> > We need to explicitly support it.
>
>    Hmm.  The inability to warp the pointer is going to put constraints
> on gui designs even outside of games.  Consider the scrollbar example,
> above.  That one isn't just a matter of locking the pointer somewhere,
> it's a matter of positioning the pointer based on the new scroll thumb
> position.  If anything we're actually better off in games in that
> scenario, because we can just shut off the pointer draw and draw a
> pointer in-engine.
>
>     I'm assuming there are sane protocol reasons for not allowing
> pointer warping, but I think you'll find it's one of those PITAs that
> you need to implement to avoid greater pain later.  Bad scroll bar
> behavior is one of those things that can grate on people.
>

Again, I don't think any of the major toolkits (Qt, GTK+, EFL..)
implement this, and I sure haven't seen it on Windows/Mac before,
so where is this concept originating from?


>     Within games, there's the classic "try to move the mouse off the
> window, the pointer stops and the map scrolls" case that we'd like to
> be able to handle.
>

That I think is a valid use case. I'm not sure how it would be implemented
(maybe with a simple "pointer barrier on this surface" request that
can be broken with AltTab and Co?).


>
> > Ah yes, deltas are the relative motion events, see above.
>
>     Deltas are quite useful, though obviously we can calculate them
> ourselves.  Some of the desire to have deltas from the system input
> comes admittedly from an admittedly somewhat childish engineering
> distaste for repeated translation back and forth between deltas and
> absolute positions as the data percolates up through the software
> stack.  Coming out of the hardware (at least for classical mice and
> trackballs) the "analog" values are all deltas.
>
> > Aah, reading this the third time, I finally understood what you meant
> > by input capture. The URL above for the relative motion events should
> > be exactly this. We are more accustomed to the term "pointer grab" or
> > "grabbing", meaning that during the grab, all input events go to this
> > particular window, until the grab is ended.
>
>     Ok, I'll try to stick to that term.  The thing is, we don't
> necessarily want *all* events routed to us; we don't want to trap
> system-level stuff like program switching (alt-tab), the "lock screen"
> button, the volume and brightness controls, the screenshot button (if
> any) and so forth.  We want *most* of the events routed to us, but not
> to the exclusion of system and window manager functionality.
>

As already mentioned, Wayland has made sure applications can't block
compositor level shortcuts and the like, so we're good here.


> > Depending on the game and physics engine, of course, is it possible to
> > make use of the input event timestamps to integrate the effect of, say,
> > a button going down some time in the past, instead of assuming it went
> > down when this game tick started?
>
>     In some games, sure.  The problem is, any lag like that can
> potentially end badly for the player.  What if we've already killed
> them before the input comes in?  What if it's a network game, and the
> new input means that instead of being killed by player B, they
> actually got player B first?
>
>     In general, the problem is that yes, we can go back and correct
> the simulation for the revised input, but what we *can't* do is revise
> the player's decisions based on the previously incorrect simulation
> that we've already showed them.  Games strive to have as tight a
> feedback loop as possible, so if the simulation is not fed input when
> it happens, we're putting information in front of the player that
> we're going to revise *after* they have started reacting to it.
>

As Pekka said, input events are dispatched once per frame, so ideally
you'd never get events more than a frame old, in which case the timestamp
might provide a small hint to the simulation. But in case of (online)
multiplayer you'd probably not constrain the simulation clock to a single
client's monitor refresh rate, so I think you have a good point that this
can be a downfall of such event delivery.

> What I'm trying to ask is, are the timestamps useful at all for games,
> > and/or would you really need a minimum latency input event delivery
> > regardless of the computational and power cost?
>
>     Timestamps can be useful as a fallback, but minimum latency is by
> far the highest priority.  Lower latency translates directly to a
> better play experience.  The difference of even a frame of lag has a
> measurable effect on player enjoyment and control.
>
> > Keeping in mind, that event based input delivery does not rely on high
> > update rates, like polling does, to not miss anything.
>
>     If the events are just coming in as a pile in 60Hz ticks, it's all
> good and we can do everything we need to.  If they're coming in as a
> pile at 10Hz ticks, it's going to be difficult to make anything more
> active than Solitaire.
>
> > There is also one more catch with the timestamps. Their base is
> > arbitrary, and a client does not know which clock produces them.
> > Therefore they are only useful as realtive to other input event
> > timestamps. Would you need a way to get the current time in the input
> > clock to be able to use them properly?
>
>     At least in our case, we're typically running the simulation off
> of either a vsync clock (consoles, mostly) or a millisecond clock
> (gettimeofday() or the platform equivalent).  Anything coming in we
> typically try to relate to those.  Some sort of timestamp we could
> relate to an actual world clock would be important; without it we'd be
> into calculating times based on heuristics, with all that implies.
>
>     VSync stamps would be good enough, or millisecond stamps.
> Anything with fixed time units.  As long as we know the size of the
> time unit and some arbitrary base time (ie: the timestamp of the first
> event we got), that's all we really need; if we need to relate it to
> the wall clock, we can call gettimeofday() and compare.  If the time
> units aren't fixed (ie: if they're just monotonically increasing IDs
> that don't actually encode time values and are only useful for
> establishing order), the results for games will be unfortunate.
>

As Pekka said, only the base is arbitrary, the value is defined as
milliseconds, so as you described you could retain one value at
simulation start and calculate deltas from thereon. The only problem
is that the timestamp will overflow and wrap around at some point,
so you would have to somehow catch that (maybe update the
the delta base at regular intervals). On the other hand, for that to
happen it'll probably take days, if not weeks.


>                                             Todd.
>
> --
>  Todd Showalter, President,
>  Electron Jump Games, Inc.
>

Jonas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/wayland-devel/attachments/20130419/12354aa3/attachment-0001.html>