Input and games.

[Todd Showalter]

On Mon, Apr 22, 2013 at 3:41 AM, Pekka Paalanen <ppaalanen at gmail.com> wrote:

> The protocol level is not convenient for an application developer in
> some cases, and it's not even meant to be. We explicitly leave lots of
> processing for a so-called toolkit library. At several points below I
> say that something is out of the scope of libwayland-client, and my
> above comment was just a fore-warning about that. :-)

    I'm aware; that said, our engine work on consoles as well, and
we're used to working close to the metal.  We may use intermediary
libraries for some things, but odds are good that for input and
graphics, at least for our engine, we'll be talking directly to (I
presume) libwayland-client.

> In other emails, it seems you are really looking for a mapping
> library for gamepads and joysticks, at least for the usual devices.
> While the Wayland protocol should support this, I do not think it
> is in scope for Wayland to actually define it. As with keyboards,
> the Wayland protocol allows passing keymaps around, and one type of
> keymaps is what xkbcommon uses. xkbcommon then actually defines the
> mappings, symbols, some state tracking, etc. which Wayland does not.

    The argument I've been trying to make is that there's a set of
gamepad functionality that is very analogous to the "standard" set of
mouse functionality.  IIRC the standard mouse functionality is
generally accepted to be defined by the USB HID spec.  My argument is
that there is a basic set of gamepad functionality (two analog sticks,
a dpad, two analog triggers, some buttons) that have become a de facto
standard such that they merit support.

    This is the kind of thing that will (hopefully) be more important
as people start building living room gaming PCs; I can't overstate the
importance of having basic game input "just work".

    I'd argue that the mouse analogy is more relevant than the
keyboard analogy; there are features present on only some keyboards
(mode shift keys, alt-gr...) that are essential to the input methods
those keyboards use.  The keyboard system needs to support a fairly
extensive set of differing hardware in order to do its job at all,
unless we're going to slink back to the 7 bit ASCII days.

    Gamepads, by contrast, are all mostly the same these days, much
like mice.  You can find oddball ones like that PC gamepad that was up
on Kickstarter recently which had a trackball in place of the right
thumb stick, but the core gamepad is now every bit as standardized as
the core mouse.

> Such a mapping library and standard should be started as a separate
> project, initially building on top of evdev directly. When that
> works, we can come up with the Wayland protocol extension to
> support that.

    Hmm.  Who would I talk to about getting this started?

>>     Which reminds me; it would be extremely useful to be able to shut
>> off key repeat for a specific client (ie: a game) without having to
>> shut it off globally.
>
> I believe key repeat is implemented client-side, so there is
> nothing to switch off. I think whether a key repeats or not depends
> also on the keymap, which the server does not process on clients'
> behalf. Instead, clients are handed the keymap and raw key values,
> and expected to do the right thing. (This is yet another thing left
> for toolkits.)

    The problem in the past here has been that in a game, you don't
typically want the keyboard to autorepeat.  In other systems (X11, for
one, IIRC) key repeat was a global setting; if you turned it off for
your program, you turned it off for everything.  So, suddenly key
repeat doesn't work in the terminal or your editor either.

    What would be nice is if there were a clean mechanism whereby a
program could say "here's how I need my input conditioned", and only
that program would be affected.

> There is actually a design principle behind having key repeat in the
> client side. Real user input always corresponds to some physical action
> of the user, happening at a point in time. The kernel input drivers
> tell us the timestamp. All Wayland protocol events corresponding to
> real user input have that timestamp. Key repeat events would not be
> real user input events.

    IIRC some autorepeat is actually hardware generated; I thought
that was something living in the BIOS, though maybe that's no longer
true.

> Furthermore, we do not fake input events. If we wanted to support e.g.
> pointer warping, we would probably need a new event to tell clients
> that the pointer was moved by something else than the user, and it
> would not have a timestamp (since we cannot assume that we can fake
> a meaningful timestamp).

    I would have thought that for pointer warping specifically that
it's one of those cases where the right thing to do is have a server
round-trip; the program with focus requests pointer warp, and the
server comes back either with a mouse move event that has been
suitably flagged, or with a warp event.  Aside from anything else,
that means:

- the warp event isn't official until it has been blessed
- the warp event can be timestamped sanely
- the server has the option to modify or reject the warp

> I think Wayland is too new to have such a general purpose library
> developed yet. Such a library should probably be formed by
> consolidating code that repeats in many toolkits, rather than be
> designed from scratch. This also assumes that there are some common
> data structures to be shared, which is not certain.

    Fair enough.  We've been building our game engine since 2003, so
I've got some admittedly strong opinions about thing like input and
low-level interfaces, but I'm not the only one working on this kind of
thing.

>>     So, yes, absolute vs. relative events is part of the issue, but
>> it's also part of a greater context; whether the keyboard is
>> generating strings or digital inputs, whether the mouse is generating
>
> FWIW, wl_keyboard generates neither, to my understanding. It sends
> some kind of keycodes, and a keymap, which describes what the codes
> mean. How you use that information is up to you.

    Ok.  Is there a policy on what happens to input during focus
changes?  For example, say I have two windows, 1 and 2, with focus on
1.  I hold down shift-a, move focus to window 2, release shift, move
focus back to window 1, release a.  What events does window 1 receive?
 How about window 2?

> The clean recovery is a major design principle in Wayland. We do
> not have a request "change videomode" at all (for normal clients).
> Instead, we have a request for: this is my window surface, please
> make it fullscreen, and I would really prefer if you
> scaled/centered/switched video mode to achieve that. That is, the
> client can make a wish of how it wants the fullscreening done, in
> case the surface size is different from the current output
> resolution. The compositor is allowed to do whatever, but of course
> it should honour the wish most of the time. The compositor knows
> the output hardware better, and it also knows everything about the
> desktop environment state (open windows, etc.), so the compositor is
> the best in charge for choosing the way. The compositor can also take
> the user's desktop preferences into account.

    Ah, good.

> That allows some interesting features. Say, your desktop is
> 1600x1200, and your game wants 1024x768. The compositor is actually
> free to change the video mode at any time it chooses. For instance,
> it can switch to 1024x768 only when the game is active and on top,
> and switch to 1600x1200 when the user activates another window. You
> could even switch between several fullscreen applications wanting
> different resolutions.

    Interesting.  I'm less sold on changing screen resolution these
days now that LCDs have largely replaced multisync monitors, but
that's still kind of neat.

> Also, should be game crash, the "video mode" is tied to the window,
> so when the window gets killed, the video mode is automatically
> restored for the desktop.

    Excellent.

> Right, and the pointer lock proposal should solve all these pointer
> problems and uses cases, but with one caveat: when the pointer lock
> is broken/released, the absolute pointer position is still where it
> was when the pointer lock was activated. So in your scroll bar
> example, the user would think the pointer warped back to the point
> where he started dragging the scroll thumb, when in reality the
> pointer never moved.

    I suppose that works, though it's not ideal.  You could presumably
do similar for things like window edge resistance.  I do think actual
warping (or at least the ability to scale pointer velocity) is
necessary in order to make effects like this look seamless.

> Btw. the functional details of the pointer lock proposal are in this patch:
> http://lists.freedesktop.org/archives/wayland-devel/2013-February/007638.html
>
> I wonder if we could solve that problem by adding a request, that
> will set the pointer position, but only during the pointer lock, and
> when the window is active, i.e. between the locked wl_pointer.enter and
> leave events. Clients only ever know about surface-local coordinates,
> so the position would be specified in surface coordinates. How would
> that sound?

    I think that could be workable.

> As for the pointer cursor, activating the pointer lock automatically
> hides the cursor surface in that proposal, so if you still need the
> cursor, you have to draw it yourself. OTOH, that should not be too
> hard, since clients always provide the cursor image, anyway.

    Ok.

> Yeah, pointer lock gives you deltas.

    That's what I would have expected, and should work fine.

> Such system level input events (e.g. hotkeys) are eaten by the
> compositor before they are ever sent to any client, so you simply
> cannot block or grab them. But, if a hotkey is actually a composition
> of several keys, like alt+tab, the client with the keyboard focus will
> see alt going down, but not the tab, IIRC. (And then the protocol takes
> care of not leaving the client with a "stuck" alt.)

    In iOS, at least at for touch stuff, they have "cancel" events,
which are basically "You know that last event I sent you?  Uhh... it
turns out it was part of a gesture, and got eaten by it.  Pretend it
never happened.".

> Btw. the input protocol is designed so, that you cannot take grabs at
> arbitrary times, and you cannot expect your grab to hold forever if you
> don't release it. The compositor is allowed to deny grab requests, and
> also break grabs at any time. It is impossible for a client to hold the
> server hostage.

    That sounds good.  As I said elsewhere, sometimes a game wants to
use the mouse in the standard way, and sometimes it wants to grab the
mouse and use it to generate deltas, but either way there has to be a
way for the game to relinquish control when the player wants to switch
to something else.  It sounds like you have that case covered.

>>     That's potentially an issue; I'd just assumed events would be
>> delivered immediately.  Might it be possible to have a knob we could
>> push to request "real time" input, for whatever value of "real time"
>> the system can manage?  Again, in some cases (mostly where a game is
>> not grabbing input) the game can deal with keyboard and mouse input at
>> desktop rates; things like text entry and gui interaction aren't
>> always time critical in games (though they can be).
>
> Right, so we should start a discussion for such a feature. The best way
> to start would be to propose the protocol for switching to "immediate
> input" mode. Another is to file a bug, so we get a permanent and
> trackable record of the feature request, if no-one wants to drive the
> protocol addition development themselves.

    I think I need to do some reading before I make proposals; I don't
know enough about Wayland yet to talk competently at the protocol
level.  That said, this is an area of interest for me, and if I can
contribute I will.

>>     If the events are just coming in as a pile in 60Hz ticks, it's all
>> good and we can do everything we need to.  If they're coming in as a
>> pile at 10Hz ticks, it's going to be difficult to make anything more
>> active than Solitaire.
>
> Yes as far as I understand, currently input events are sent to Wayland
> clients as a burst at every compositor repaint cycle, which happens at
> the monitor refresh rate, so for a 60 Hz monitor, you would be getting
> them in bursts at 60 Hz.

    That's as good as we get from consoles, and besides, there isn't
much point in most games in running your simulator at a higher
frequency than the refresh rate.

> Note, that this is specific to Weston, though. I'm not sure if we have
> any guidelines on this for other compositors, and the protocol does not
> say anything about it. I cannot imagine why anyone would want to delay
> input events beyond the repaint cycle. Others might just do the
> immediate sending, since it is easier to implement.

    I could see a network-transparent client holding events for
longer, though at that point it might be the least of your worries if
you're running a game on it.

> The timestamps are specified to be in milliseconds, and preferrably from
> a monotonic clock (which e.g. gettimeofday is not). The things that are
> not specified are the absolute value of the clock (or epoch, or
> relation to the real time clock), and resolution.

    That's still probably good enough.  If you've got input data sets
coming in at the frame rate of the display, and you know that the data
is internally ordered, I think there's enough to work with.  We're
essentially talking about an ordered set of deltas or positions
covering a 16.67ms snapshot of time.  The timeframe isn't long enough
that the intersample time provides any useful nuance, at least not
with input from humans.

> Matching input event timestamps to any specific clock you read yourself
> is not easy. The input event timestamp is at an unknown time in the past,
> while the clock you read is "current". I can't see a way to just
> compute the difference.

    If I know what frame it came in on, and I know what gettimeofday()
at the beginning of that frame update told me, it's good enough for my
purposes.  With more mechanical things it's generally safe to assume
the data is evenly spaced across the duration of the timeslice it came
from.

> Therefore we would need a protocol extension to ask in which clock do
> the timestamps come, and still the answer might be something you do not
> know about, or cannot access, so you would still need some fallback.
> And we cannot just ask the compositor for the current time, since the
> compositor might not have access to that clock, and there is a delay
> between the compositor reading the clock, and the client waking up and
> processing the event.

    I'm not actually totally sure it would be needed, based on what
was said above.

> Btw. the vsync and presentation events, and their timestamps are yet
> another whole new story, unrelated to input. :-)

    Indeed.  Not to mention synchronizing multiple windows running
different views of the same game.

                                           Todd.

--
 Todd Showalter, President,
 Electron Jump Games, Inc.