Input and games.

[Pekka Paalanen]

On Mon, 22 Apr 2013 10:42:35 -0400
Todd Showalter <todd at electronjump.com> wrote:

> On Mon, Apr 22, 2013 at 3:41 AM, Pekka Paalanen <ppaalanen at gmail.com> wrote:
> 
> > The protocol level is not convenient for an application developer in
> > some cases, and it's not even meant to be. We explicitly leave lots of
> > processing for a so-called toolkit library. At several points below I
> > say that something is out of the scope of libwayland-client, and my
> > above comment was just a fore-warning about that. :-)
> 
>     I'm aware; that said, our engine work on consoles as well, and
> we're used to working close to the metal.  We may use intermediary
> libraries for some things, but odds are good that for input and
> graphics, at least for our engine, we'll be talking directly to (I
> presume) libwayland-client.
> 
> > In other emails, it seems you are really looking for a mapping
> > library for gamepads and joysticks, at least for the usual devices.
> > While the Wayland protocol should support this, I do not think it
> > is in scope for Wayland to actually define it. As with keyboards,
> > the Wayland protocol allows passing keymaps around, and one type of
> > keymaps is what xkbcommon uses. xkbcommon then actually defines the
> > mappings, symbols, some state tracking, etc. which Wayland does not.
> 
>     The argument I've been trying to make is that there's a set of
> gamepad functionality that is very analogous to the "standard" set of
> mouse functionality.  IIRC the standard mouse functionality is
> generally accepted to be defined by the USB HID spec.  My argument is
> that there is a basic set of gamepad functionality (two analog sticks,
> a dpad, two analog triggers, some buttons) that have become a de facto
> standard such that they merit support.
> 
>     This is the kind of thing that will (hopefully) be more important
> as people start building living room gaming PCs; I can't overstate the
> importance of having basic game input "just work".
> 
>     I'd argue that the mouse analogy is more relevant than the
> keyboard analogy; there are features present on only some keyboards
> (mode shift keys, alt-gr...) that are essential to the input methods
> those keyboards use.  The keyboard system needs to support a fairly
> extensive set of differing hardware in order to do its job at all,
> unless we're going to slink back to the 7 bit ASCII days.
> 
>     Gamepads, by contrast, are all mostly the same these days, much
> like mice.  You can find oddball ones like that PC gamepad that was up
> on Kickstarter recently which had a trackball in place of the right
> thumb stick, but the core gamepad is now every bit as standardized as
> the core mouse.

Alright, do you really mean that the controls are as standard as
mouse buttons and wheels, and we would not need a per-device-model
database? If so, then sure, a mouse-like Wayland protocol would
indeed be possible.

> > Such a mapping library and standard should be started as a separate
> > project, initially building on top of evdev directly. When that
> > works, we can come up with the Wayland protocol extension to
> > support that.
> 
>     Hmm.  Who would I talk to about getting this started?

I'm not sure. If you're looking for volunteers, just throwing the
idea out in public is a start, but to have some chances of
succeeding, you probably need to start the work yourself, or pay
someone to do it. If it turns out good, other projects might start
using it, and also contributing.

But as per above, maybe we really don't need it?

> >>     Which reminds me; it would be extremely useful to be able to shut
> >> off key repeat for a specific client (ie: a game) without having to
> >> shut it off globally.
> >
> > I believe key repeat is implemented client-side, so there is
> > nothing to switch off. I think whether a key repeats or not depends
> > also on the keymap, which the server does not process on clients'
> > behalf. Instead, clients are handed the keymap and raw key values,
> > and expected to do the right thing. (This is yet another thing left
> > for toolkits.)
> 
>     The problem in the past here has been that in a game, you don't
> typically want the keyboard to autorepeat.  In other systems (X11, for
> one, IIRC) key repeat was a global setting; if you turned it off for
> your program, you turned it off for everything.  So, suddenly key
> repeat doesn't work in the terminal or your editor either.
> 
>     What would be nice is if there were a clean mechanism whereby a
> program could say "here's how I need my input conditioned", and only
> that program would be affected.

Looking at Weston, it seems to do the extra effort to ensure that
it does not send repeats to clients.

> > There is actually a design principle behind having key repeat in the
> > client side. Real user input always corresponds to some physical action
> > of the user, happening at a point in time. The kernel input drivers
> > tell us the timestamp. All Wayland protocol events corresponding to
> > real user input have that timestamp. Key repeat events would not be
> > real user input events.
> 
>     IIRC some autorepeat is actually hardware generated; I thought
> that was something living in the BIOS, though maybe that's no longer
> true.

Maybe, but I would tend to think that native device drivers kick
such BIOS features out, when they initialize.

> > Furthermore, we do not fake input events. If we wanted to support e.g.
> > pointer warping, we would probably need a new event to tell clients
> > that the pointer was moved by something else than the user, and it
> > would not have a timestamp (since we cannot assume that we can fake
> > a meaningful timestamp).
> 
>     I would have thought that for pointer warping specifically that
> it's one of those cases where the right thing to do is have a server
> round-trip; the program with focus requests pointer warp, and the
> server comes back either with a mouse move event that has been
> suitably flagged, or with a warp event.  Aside from anything else,
> that means:
> 
> - the warp event isn't official until it has been blessed
> - the warp event can be timestamped sanely
> - the server has the option to modify or reject the warp

Still, I think would be problematic. If a client continuously warps
the pointer to the middle of its window, getting away from that
would be difficult, and I can't see any heuristic the compositor
could use to prevent that.

Granted, it's not that different from pointer lock. I just believe
that arbitrary pointer warping is disrupting to a user, and we need
to limit it to special cases, like pointer lock. Even just
requiring keyboard focus to be able to warp goes a long way.

Let's see if we could solve the issues better than relying on
general pointer warping.

> > I think Wayland is too new to have such a general purpose library
> > developed yet. Such a library should probably be formed by
> > consolidating code that repeats in many toolkits, rather than be
> > designed from scratch. This also assumes that there are some common
> > data structures to be shared, which is not certain.
> 
>     Fair enough.  We've been building our game engine since 2003, so
> I've got some admittedly strong opinions about thing like input and
> low-level interfaces, but I'm not the only one working on this kind of
> thing.

It is always good to have professional opinions. I'm not sure if we
have had any game developers looking into things before you.

> >>     So, yes, absolute vs. relative events is part of the issue, but
> >> it's also part of a greater context; whether the keyboard is
> >> generating strings or digital inputs, whether the mouse is generating
> >
> > FWIW, wl_keyboard generates neither, to my understanding. It sends
> > some kind of keycodes, and a keymap, which describes what the codes
> > mean. How you use that information is up to you.
> 
>     Ok.  Is there a policy on what happens to input during focus
> changes?  For example, say I have two windows, 1 and 2, with focus on
> 1.  I hold down shift-a, move focus to window 2, release shift, move
> focus back to window 1, release a.  What events does window 1 receive?
>  How about window 2?

Yes. This is from the top of my head. When a window loses keyboard
focus, it should just stop doing anything with the last keyboard
state it got, since it doesn't have the keyboard anymore. A window
that receives keyboard focus should reset its keyboard state, and
use the new initial state transmitted with the keyboard enter event.

http://wayland.freedesktop.org/docs/html/protocol-spec-interface-wl_keyboard.html

Keyboard/pointer enter and leave events carry input state with
them, but they are not directly caused by real user input, and so
do not have e.g. a timestamp. If there are real user actions
involved, they will come as separate events.

An example of receiving keyboard focus, i.e. enter event, not due
to a user action is a program closing a window that had keyboard
focus, and focus moving to the next window.

> > The clean recovery is a major design principle in Wayland. We do
> > not have a request "change videomode" at all (for normal clients).
> > Instead, we have a request for: this is my window surface, please
> > make it fullscreen, and I would really prefer if you
> > scaled/centered/switched video mode to achieve that. That is, the
> > client can make a wish of how it wants the fullscreening done, in
> > case the surface size is different from the current output
> > resolution. The compositor is allowed to do whatever, but of course
> > it should honour the wish most of the time. The compositor knows
> > the output hardware better, and it also knows everything about the
> > desktop environment state (open windows, etc.), so the compositor is
> > the best in charge for choosing the way. The compositor can also take
> > the user's desktop preferences into account.
> 
>     Ah, good.
> 
> > That allows some interesting features. Say, your desktop is
> > 1600x1200, and your game wants 1024x768. The compositor is actually
> > free to change the video mode at any time it chooses. For instance,
> > it can switch to 1024x768 only when the game is active and on top,
> > and switch to 1600x1200 when the user activates another window. You
> > could even switch between several fullscreen applications wanting
> > different resolutions.
> 
>     Interesting.  I'm less sold on changing screen resolution these
> days now that LCDs have largely replaced multisync monitors, but
> that's still kind of neat.

Indeed, but you can still choose between GPU and panel scaling,
should scaling be needed. And of course centering without scaling.
GPU scaling may or may not imply a composition operation, which is
essentially a copy. The other options can be zero-copy paths, if
the client provides buffers suitable for scanout (we achieve that
basically via EGL).

> > Also, should be game crash, the "video mode" is tied to the window,
> > so when the window gets killed, the video mode is automatically
> > restored for the desktop.
> 
>     Excellent.
> 
> > Right, and the pointer lock proposal should solve all these pointer
> > problems and uses cases, but with one caveat: when the pointer lock
> > is broken/released, the absolute pointer position is still where it
> > was when the pointer lock was activated. So in your scroll bar
> > example, the user would think the pointer warped back to the point
> > where he started dragging the scroll thumb, when in reality the
> > pointer never moved.
> 
>     I suppose that works, though it's not ideal.  You could presumably
> do similar for things like window edge resistance.  I do think actual
> warping (or at least the ability to scale pointer velocity) is
> necessary in order to make effects like this look seamless.
> 
> > Btw. the functional details of the pointer lock proposal are in this patch:
> > http://lists.freedesktop.org/archives/wayland-devel/2013-February/007638.html
> >
> > I wonder if we could solve that problem by adding a request, that
> > will set the pointer position, but only during the pointer lock, and
> > when the window is active, i.e. between the locked wl_pointer.enter and
> > leave events. Clients only ever know about surface-local coordinates,
> > so the position would be specified in surface coordinates. How would
> > that sound?
> 
>     I think that could be workable.

Alright, I'll try to keep that in mind when the pointer lock comes
up the next time.

> > As for the pointer cursor, activating the pointer lock automatically
> > hides the cursor surface in that proposal, so if you still need the
> > cursor, you have to draw it yourself. OTOH, that should not be too
> > hard, since clients always provide the cursor image, anyway.
> 
>     Ok.
> 
> > Yeah, pointer lock gives you deltas.
> 
>     That's what I would have expected, and should work fine.
> 
> > Such system level input events (e.g. hotkeys) are eaten by the
> > compositor before they are ever sent to any client, so you simply
> > cannot block or grab them. But, if a hotkey is actually a composition
> > of several keys, like alt+tab, the client with the keyboard focus will
> > see alt going down, but not the tab, IIRC. (And then the protocol takes
> > care of not leaving the client with a "stuck" alt.)
> 
>     In iOS, at least at for touch stuff, they have "cancel" events,
> which are basically "You know that last event I sent you?  Uhh... it
> turns out it was part of a gesture, and got eaten by it.  Pretend it
> never happened.".

Yes, touch protocols are quite different to conventional input
protocols, and I'm not too familiar with them.

> > Btw. the input protocol is designed so, that you cannot take grabs at
> > arbitrary times, and you cannot expect your grab to hold forever if you
> > don't release it. The compositor is allowed to deny grab requests, and
> > also break grabs at any time. It is impossible for a client to hold the
> > server hostage.
> 
>     That sounds good.  As I said elsewhere, sometimes a game wants to
> use the mouse in the standard way, and sometimes it wants to grab the
> mouse and use it to generate deltas, but either way there has to be a
> way for the game to relinquish control when the player wants to switch
> to something else.  It sounds like you have that case covered.

Yes, it should be covered with the pointer lock proposal.

> >>     That's potentially an issue; I'd just assumed events would be
> >> delivered immediately.  Might it be possible to have a knob we could
> >> push to request "real time" input, for whatever value of "real time"
> >> the system can manage?  Again, in some cases (mostly where a game is
> >> not grabbing input) the game can deal with keyboard and mouse input at
> >> desktop rates; things like text entry and gui interaction aren't
> >> always time critical in games (though they can be).
> >
> > Right, so we should start a discussion for such a feature. The best way
> > to start would be to propose the protocol for switching to "immediate
> > input" mode. Another is to file a bug, so we get a permanent and
> > trackable record of the feature request, if no-one wants to drive the
> > protocol addition development themselves.
> 
>     I think I need to do some reading before I make proposals; I don't
> know enough about Wayland yet to talk competently at the protocol
> level.  That said, this is an area of interest for me, and if I can
> contribute I will.

We could start with just a feature request bug report, but from the
below it sounds like you would not really need this specific feature
after all.

> >>     If the events are just coming in as a pile in 60Hz ticks, it's all
> >> good and we can do everything we need to.  If they're coming in as a
> >> pile at 10Hz ticks, it's going to be difficult to make anything more
> >> active than Solitaire.
> >
> > Yes as far as I understand, currently input events are sent to Wayland
> > clients as a burst at every compositor repaint cycle, which happens at
> > the monitor refresh rate, so for a 60 Hz monitor, you would be getting
> > them in bursts at 60 Hz.
> 
>     That's as good as we get from consoles, and besides, there isn't
> much point in most games in running your simulator at a higher
> frequency than the refresh rate.

What about these hardcore fps-gamers who want at least 200+ frames
per second, or they can't win? :-)

That is something I have never understood, apart from game engine
bugs where frame rate affected the physics simulation outcome like
allowing to jump higher.

> > Note, that this is specific to Weston, though. I'm not sure if we have
> > any guidelines on this for other compositors, and the protocol does not
> > say anything about it. I cannot imagine why anyone would want to delay
> > input events beyond the repaint cycle. Others might just do the
> > immediate sending, since it is easier to implement.
> 
>     I could see a network-transparent client holding events for
> longer, though at that point it might be the least of your worries if
> you're running a game on it.

Heh (you mean server?).

> > The timestamps are specified to be in milliseconds, and preferrably from
> > a monotonic clock (which e.g. gettimeofday is not). The things that are
> > not specified are the absolute value of the clock (or epoch, or
> > relation to the real time clock), and resolution.
> 
>     That's still probably good enough.  If you've got input data sets
> coming in at the frame rate of the display, and you know that the data
> is internally ordered, I think there's enough to work with.  We're
> essentially talking about an ordered set of deltas or positions
> covering a 16.67ms snapshot of time.  The timeframe isn't long enough
> that the intersample time provides any useful nuance, at least not
> with input from humans.

Are you sure about that? Sorry for insisting, but this is the first
time I can ask a game professional about that. :-)

> > Matching input event timestamps to any specific clock you read yourself
> > is not easy. The input event timestamp is at an unknown time in the past,
> > while the clock you read is "current". I can't see a way to just
> > compute the difference.
> 
>     If I know what frame it came in on, and I know what gettimeofday()
> at the beginning of that frame update told me, it's good enough for my
> purposes.  With more mechanical things it's generally safe to assume
> the data is evenly spaced across the duration of the timeslice it came
> from.
> 
> > Therefore we would need a protocol extension to ask in which clock do
> > the timestamps come, and still the answer might be something you do not
> > know about, or cannot access, so you would still need some fallback.
> > And we cannot just ask the compositor for the current time, since the
> > compositor might not have access to that clock, and there is a delay
> > between the compositor reading the clock, and the client waking up and
> > processing the event.
> 
>     I'm not actually totally sure it would be needed, based on what
> was said above.

Alright.

> > Btw. the vsync and presentation events, and their timestamps are yet
> > another whole new story, unrelated to input. :-)
> 
>     Indeed.  Not to mention synchronizing multiple windows running
> different views of the same game.


Thanks,
pq