<div dir="ltr">2013/4/19 Todd Showalter <span dir="ltr"><<a href="mailto:todd@electronjump.com" target="_blank">todd@electronjump.com</a>></span><br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Fri, Apr 19, 2013 at 5:18 AM, Pekka Paalanen <<a href="mailto:ppaalanen@gmail.com">ppaalanen@gmail.com</a>> wrote:<br>
</div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"><br></div><div class="im">
> Event driven is a little more work for the "simple" games, but it gives<br>
> you guarantees. Would you not agree?<br>
<br>
</div> We can definitely work with it. As much as anything it's a<br>
question of convenience; the question is really how much<br>
superstructure we need to build on top to get what we need. We've<br>
already got that superstructure elsewhere, so porting it over is<br>
simple enough. It would be more convenient if we didn't have to, but<br>
it's not a deal breaker.<br>
<br>
For context, I'm not trying to convince you to change the protocol<br>
or the model per se; aside from anything else, I don't yet understand<br>
it well enough to seriously critique it. A large part of what I'm<br>
hoping to do here is offer some insight into how games tend to use<br>
input, the kind of needs games often have, and the sorts of<br>
considerations that make a system easier or harder to put a game on.<br>
Wayland obviously has competing considerations, some of which are<br>
arguably more important than games. If one can imagine such a thing.<br>
<br>
One thing worth noting here is why we want operate on virtualized<br>
input structures rather than raw events. One reason I mentioned<br>
above; accumulating events so that they can be applied between frames.<br>
Another reason is complexity management; games can be quite complex<br>
beasts consisting of many parts, and everything that can be done to<br>
isolate those parts makes the game easier to develop and maintain.<br>
<br>
The classic problem with a purely event-driven program is that<br>
somewhere in it there is a giant event loop that knows about<br>
everything in the program. In something simple like a calculator,<br>
it's not a problem, but once you scale up to a large system with<br>
multiple subsystems the event loop can turn into a nightmare. Having<br>
virtualized input structures that the game can query means that input<br>
tests can be isolated to the code where they belong. ie:<br>
<br>
if(KeyTrigger(KEY_D) && KeyDown(KEY_CTRL))<br>
{<br>
Log("heap integrity %d\n", check_heap_integrity());<br>
}<br>
<br>
You can achieve some of the same modularity with function pointer<br>
lists or similar hooks, but having a virtualized input structure has<br>
(in my experience at least) been the cleanest abstraction. </blockquote><div><br></div><div>I can totally see where you're coming from (having worked on a small<br></div><div>engine myself in the past), but I feel like creating static input states<br>
</div><div>should always be done on client side. Especially in Wayland where<br></div><div>"frame-perfectness" is crucial, round-trips such as input state polling<br></div><div>are strongly discouraged.<br></div>
<div>On the other hand, this doesn't mean that every developer has to<br>reinvent the wheel. Input state caching could certainly be split off<br></div><div>into a client library.<br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
> Is this referring to the problem of "oops, my mouse left the Quake<br>
> window when I tried to turn"? Or maybe more of "oops, the pointer hit<br>
> the monitor edge and I cannot turn any more?" I.e. absolute vs.<br>
> relative input events?<br>
<br>
</div> Partly. The issue is that *sometimes* a game wants the mouse and<br>
keyboard to behave in the standard way (ie: the mouse controls the<br>
pointer and lets you click gui elements, the keyboard is for entering<br>
text and hitting control keys) and *sometimes* the game wants the<br>
mouse motion to control an in-game object (often the camera) and just<br>
wants the keyboard and mouse buttons to be a big bag of digital<br>
buttons. With the Quake example, when the pause menu is up, or when<br>
the terminal has been called down, the game wants the keyboard to be<br>
generating text commands on the terminal and the mouse to be able to<br>
select text and click on buttons. When the terminal is gone and the<br>
game isn't paused, Quake wants the mouse to control the camera view<br>
and the keyboard WASD keys are emulating a game controller dpad.<br>
<br>
So, yes, absolute vs. relative events is part of the issue, but<br>
it's also part of a greater context; whether the keyboard is<br>
generating strings or digital inputs, whether the mouse is generating<br>
positions or deltas, under what circumstances focus is allowed to<br>
leave the window, whether the mouse pointer is visible, and things<br>
like how system-wide hotkeys factor in to things. Can I capture the<br>
keyboard and mouse without preventing the user from using alt-tab to<br>
switch to another program, for instance?<br></blockquote><div><br></div><div>Well, at least with the "relative motion" proposal Pekka linked,<br>such a grab would always be breakable by things such as AltTab.<br>
</div><div>AFAIK the general consensus is that Wayland clients should never ever<br>be able to "hold the display server hostage" as would often happen in X11<br>with faulty clients (ie. unable to leave a fullscreen hanging game).<br>
</div><div>So you shouldn't worry about that.<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Clean, fast switching between these states is part of it as well;<br>
in a game like Quake, as above, you want to be able to capture the<br>
mouse when the game is playing, but "uncapture" it when the pause menu<br>
or the game terminal are up, or if the player switches focus to<br>
another program. In an RTS, you might want a visible cursor but want<br>
to constrain the mouse to the window to allow the map to scroll. You<br>
might want to use the keyboard mostly for hotkeys, but if they hit<br>
enter you want them to be able to type a string in to broadcast to<br>
multiplayer chat. The scroll wheel might control either the message<br>
scrollback or the zoom level, depending on what the cursor is floating<br>
over.<br></blockquote><div><br></div><div>I think the biggest reason for "relative motion" is to prevent having to warp<br>the pointer, which wayland won't support. Everything else, ie. how your<br></div><div>
client interprets keyboard/scroll events, is up to you. I don't think it's<br>wise to let compositors send "text" events, that's the reason the keymap is<br>provided as a fd. And it's certainly not a good idea to tell them to suddenly<br>
</div><div>provide different events for the same physical buttons/keys.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
There's also the question of clean recovery; if a game has changed<br>
the video mode (if that's allowed any more, though these days with LCD<br>
panels and robust 3D hardware maybe that's just a bad idea), turned<br>
off key repeat and captured the mouse, all of that needs to be<br>
reverted if the game exits ungracefully. Which sometimes happens,<br>
especially during development.<br></blockquote><div><br></div><div>True. But as I mentioned above, what is even more critical than that is the<br>ability to escape _hanged_ clients without having to reboot your PC.<br>I think in that area the wayland devs are already cautious about doing the<br>
right thing. Global (compositor) hotkeys/shortcuts will never be able to<br>be swallowed by clients AFAIK.<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
> There is a relative motion events proposal for mice:<br>
> <a href="http://lists.freedesktop.org/archives/wayland-devel/2013-February/007635.html" target="_blank">http://lists.freedesktop.org/archives/wayland-devel/2013-February/007635.html</a><br>
<br>
</div> Something like that will be needed for a lot of styles of game,<br>
but also has use elsewhere. For example, there used to be a widget on<br>
Irix machines IIRC, that looked like a trackball. If you put the<br>
mouse pointer on it, held down the mouse button and then moved the<br>
mouse, it would scroll the trackball control rather than move the<br>
mouse pointer.<br>
<br>
Similarly, when you're doing scroll bars, if you want to make a<br>
scroll bar where dragging the thumb moves the scrolled view at a rate<br>
that is pixel-proportional rather than window-size proportional, you<br>
have to be able to warp the pointer; otherwise, the view is slaved to<br>
the thumb position, so taller views scroll by faster.<br>
<br>
Concrete example: Let's say I have a document that is 1000 pixels<br>
tall, in a view that's 400 pixels tall. Let's fudge the math a bit,<br>
say the thumb is one pixel tall and the region the thumb can be<br>
scrolled over is the full height of the window. The window shows 40%<br>
of the document. Without pointer warping, each step in the scroll bar<br>
is (600 / 400) pixels, so we're scrolling on average 1.5 pixels of<br>
view for every pixel the thumb moves up or down the screen.<br>
<br>
Now, in the same view, we have a 250000 pixel tall document. The<br>
document got longer, but the scroll bar is the same height (and thus,<br>
the same number of steps). Each step of the scroll bar is now (249600<br>
/ 400), or 624 pixels, enough that each scroll thumb movement scrolls<br>
more than 1.5x the view area.<br>
<br>
The classic solution to this is when the scroll amount goes above<br>
or below sane thresholds, the view is moved by a sane amount, the<br>
scroll bar is moved by the correct amount (if any) for the new view,<br>
and if necessary the pointer is warped to the new thumb position.<br>
</blockquote><div><br></div><div>Ok, now this seems REALLY confusing to me. In over 10 years of using<br></div><div>computers I have never come across a single application behaving like that.<br></div><div>Could you maybe name an example? Also, isn't this the reason mouse<br>
wheels were invented in the first place? I can see where you're coming<br>from, but for me personally scroll bars have always been more of an<br></div><div>"absolute" control widget, ie. I use them when I want to go exactly to<br>
the beginning/end/40% mark of a viewport.<br></div><div>Also, I'm pretty sure something like mouse warping will likely never be<br>implemented in wayland for design reasons (you don't want malicious<br>apps controlling crucial variables such as pointer location). <br>
</div><div><br> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
> Clients cannot warp the pointer, so there is no way to hack around it.<br>
> We need to explicitly support it.<br>
<br>
</div> Hmm. The inability to warp the pointer is going to put constraints<br>
on gui designs even outside of games. Consider the scrollbar example,<br>
above. That one isn't just a matter of locking the pointer somewhere,<br>
it's a matter of positioning the pointer based on the new scroll thumb<br>
position. If anything we're actually better off in games in that<br>
scenario, because we can just shut off the pointer draw and draw a<br>
pointer in-engine.<br>
<br>
I'm assuming there are sane protocol reasons for not allowing<br>
pointer warping, but I think you'll find it's one of those PITAs that<br>
you need to implement to avoid greater pain later. Bad scroll bar<br>
behavior is one of those things that can grate on people.<br></blockquote><div><br></div><div>Again, I don't think any of the major toolkits (Qt, GTK+, EFL..)<br>implement this, and I sure haven't seen it on Windows/Mac before,<br>
so where is this concept originating from?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Within games, there's the classic "try to move the mouse off the<br>
window, the pointer stops and the map scrolls" case that we'd like to<br>
be able to handle.<br></blockquote><div><br></div><div>That I think is a valid use case. I'm not sure how it would be implemented<br></div><div>(maybe with a simple "pointer barrier on this surface" request that<br>
can be broken with AltTab and Co?).<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
> Ah yes, deltas are the relative motion events, see above.<br>
<br>
</div> Deltas are quite useful, though obviously we can calculate them<br>
ourselves. Some of the desire to have deltas from the system input<br>
comes admittedly from an admittedly somewhat childish engineering<br>
distaste for repeated translation back and forth between deltas and<br>
absolute positions as the data percolates up through the software<br>
stack. Coming out of the hardware (at least for classical mice and<br>
trackballs) the "analog" values are all deltas.<br>
<div class="im"><br>
> Aah, reading this the third time, I finally understood what you meant<br>
> by input capture. The URL above for the relative motion events should<br>
> be exactly this. We are more accustomed to the term "pointer grab" or<br>
> "grabbing", meaning that during the grab, all input events go to this<br>
> particular window, until the grab is ended.<br>
<br>
</div> Ok, I'll try to stick to that term. The thing is, we don't<br>
necessarily want *all* events routed to us; we don't want to trap<br>
system-level stuff like program switching (alt-tab), the "lock screen"<br>
button, the volume and brightness controls, the screenshot button (if<br>
any) and so forth. We want *most* of the events routed to us, but not<br>
to the exclusion of system and window manager functionality.<br></blockquote><div><br></div><div>As already mentioned, Wayland has made sure applications can't block<br></div><div>compositor level shortcuts and the like, so we're good here.<br>
<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"><br>
> Depending on the game and physics engine, of course, is it possible to<br>
> make use of the input event timestamps to integrate the effect of, say,<br>
> a button going down some time in the past, instead of assuming it went<br>
> down when this game tick started?<br>
<br>
</div> In some games, sure. The problem is, any lag like that can<br>
potentially end badly for the player. What if we've already killed<br>
them before the input comes in? What if it's a network game, and the<br>
new input means that instead of being killed by player B, they<br>
actually got player B first?<br>
<br>
In general, the problem is that yes, we can go back and correct<br>
the simulation for the revised input, but what we *can't* do is revise<br>
the player's decisions based on the previously incorrect simulation<br>
that we've already showed them. Games strive to have as tight a<br>
feedback loop as possible, so if the simulation is not fed input when<br>
it happens, we're putting information in front of the player that<br>
we're going to revise *after* they have started reacting to it.<br></blockquote><div><br></div><div>As Pekka said, input events are dispatched once per frame, so ideally<br>you'd never get events more than a frame old, in which case the timestamp<br>
</div><div>might provide a small hint to the simulation. But in case of (online)<br></div><div>multiplayer you'd probably not constrain the simulation clock to a single<br>client's monitor refresh rate, so I think you have a good point that this<br>
can be a downfall of such event delivery.<br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
> What I'm trying to ask is, are the timestamps useful at all for games,<br>
> and/or would you really need a minimum latency input event delivery<br>
> regardless of the computational and power cost?<br>
<br>
</div> Timestamps can be useful as a fallback, but minimum latency is by<br>
far the highest priority. Lower latency translates directly to a<br>
better play experience. The difference of even a frame of lag has a<br>
measurable effect on player enjoyment and control.<br>
<div class="im"><br>
> Keeping in mind, that event based input delivery does not rely on high<br>
> update rates, like polling does, to not miss anything.<br>
<br>
</div> If the events are just coming in as a pile in 60Hz ticks, it's all<br>
good and we can do everything we need to. If they're coming in as a<br>
pile at 10Hz ticks, it's going to be difficult to make anything more<br>
active than Solitaire.<br>
<div class="im"><br>
> There is also one more catch with the timestamps. Their base is<br>
> arbitrary, and a client does not know which clock produces them.<br>
> Therefore they are only useful as realtive to other input event<br>
> timestamps. Would you need a way to get the current time in the input<br>
> clock to be able to use them properly?<br>
<br>
</div> At least in our case, we're typically running the simulation off<br>
of either a vsync clock (consoles, mostly) or a millisecond clock<br>
(gettimeofday() or the platform equivalent). Anything coming in we<br>
typically try to relate to those. Some sort of timestamp we could<br>
relate to an actual world clock would be important; without it we'd be<br>
into calculating times based on heuristics, with all that implies.<br>
<br>
VSync stamps would be good enough, or millisecond stamps.<br>
Anything with fixed time units. As long as we know the size of the<br>
time unit and some arbitrary base time (ie: the timestamp of the first<br>
event we got), that's all we really need; if we need to relate it to<br>
the wall clock, we can call gettimeofday() and compare. If the time<br>
units aren't fixed (ie: if they're just monotonically increasing IDs<br>
that don't actually encode time values and are only useful for<br>
establishing order), the results for games will be unfortunate.<br></blockquote><div><br></div><div>As Pekka said, only the base is arbitrary, the value is defined as<br>milliseconds, so as you described you could retain one value at<br>
</div><div>simulation start and calculate deltas from thereon. The only problem<br></div><div>is that the timestamp will overflow and wrap around at some point,<br>so you would have to somehow catch that (maybe update the<br>
</div><div>the delta base at regular intervals). On the other hand, for that to<br>happen it'll probably take days, if not weeks.<br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5"><br>
Todd.<br>
<br>
--<br>
Todd Showalter, President,<br>
Electron Jump Games, Inc.<br>
</div></div></blockquote></div><br></div><div class="gmail_extra">Jonas<br></div></div>