Gamepad focus model (Re: Input and games.)

[Rick Yorgason]

On 2013-05-14 11:18, Daniel Stone wrote:
>> 1) Some compositors may put every gamepad in one seat, and some compositors
>> may assign one gamepad per seat. The game programmer has to support both
>> code paths. And unfortunately, when game programmers inevitably neglect to
>> do that, the gamers are going to blame the compositors rather than the
>> games. Wayland will get a reputation as a display server where you have to
>> configure your gamepads differently for different games.
>
> Compositors can implement maximise badly too, or multi-display
> support.  But it's not an argument to push the complexity out.  It's
> an argument for not running broken compositors.

But is it really broken?

The *only* way to do a split screeen FPS where both players are using a 
keyboard/mouse combo is to use multiple seats. That alone should be an 
argument for putting the player ID in the seat rather than the gamepad.

>> 2) Gamepads aren't the only scenario where you want to have one focus for
>> multiple players.
>>
>> Here's a couple scenarios that are complicated by breaking the seat == user
>> model:
>
> It's not seat == user, it's seat == focus.

Well, right now we have user == seat == focus. We're debating which side 
of the equation should be decoupled.

>> A) You're playing split-screen Halo. Two users have a mouse/keyboard, and
>> two users have gamepads.
>>
>> In the seat == user world, we know how this is set up: seat 1 has a
>> mouse/keyboard, seat 2 has a mouse/keyboard, seat 3 has a gamepad, and seat
>> 4 has a gamepad.
>
> We know how it's set up, but we don't know how it works.  How does the
> compositor know to focus 3 and 4 at Halo, rather than at the browser
> or whatever else is running at the time? You can't have the keyboard
> and mouse tell you, because they're different seats.  So it's just
> manual then? Or the compositor making random guesses?

Are the details of how focus slaving should work the only thing holding 
us back from this model?

I've given some examples of how I think a compositor should implement 
that in earlier emails, but I can work on explaining it better if it's 
the only problem.

>> In the many-gamepads-per-seat model, I don't know how to set this up. The
>> first two players would have to have their own seats, but since the player
>> ID is in the gamepad, where do we get their player numbers from? And what
>> about players 3 and 4? Where do their gamepads go? They can go in seat 1,
>> but then how do you distinguish player 1 from players 3 and 4? And why does
>> player 2 have to manually focus the game?
>
> The gamepads go to whichever seat it is that's directing their focus.
> If their focus is entirely manually set, then they go in their own
> seat.  Simple, no? Or am I totally missing something, and do gamepads
> under Wayland typically not follow keyboard/mouse focii? (I don't mean
> in really contrived theoretical cases, I mean in the overwhelming
> majority of situations.)

Okay, so in this scenario, seat 1 has a mouse, a keyboard, and two 
gamepads, and those devices are split across players 1, 3, and 4.

The gamepad object we discussed earlier had a player ID associated with 
it, so you could tell which gamepad belongs to player 1, player 2, etc. 
If the gamepad has player indicator LEDs, it's expected to display this 
information.

But in this case, player 1 has the mouse and keyboard, and there's no 
way to represent that. Instead, the compositor would probably assign the 
gamepads to players 1 and 2, but since they're *actually* players 3 and 
4, the LEDs on their gamepads would be wrong.

I *do* believe that gamepads in Wayland would typically follow the 
keyboard focus, but I think that can be accomplished with focus slaving, 
without any changes to the protocol.

>> (I don't believe this scenario is inherently exotic. Lemmings, Settlers, and
>> Hired Guns all did this back in the DOS and Amiga days, before OSs made it
>> harder to distinguish mice from each other.)
>
> Yes, that's the tradeoff you get for multitasking, and having the OS
> know where your mice are, rather than having to tell every single game
> where every single mouse is.

Yep, it's a tradeoff, but only because display servers haven't been 
designed with multiple simultaneous users in mind. This is one of the 
many areas Wayland innovates in.

>> B) Imagine that, with the current trend of adding touchpads to gamepads,
>> Fruit Ninja releases a multi-player version. All players play on the same
>> "field", but it tracks their scores independently; whomever slashes the
>> fruit first gets the points. Let's imagine we have player 1 with a separate
>> touchpad, and players 2 and 3 with gamepads that have touchpads built in.
>>
>> In the seat == user model, it's obvious how this should be set up: seat 1
>> gets a touchpad, seat 2 gets a gamepad with the associated touchpad, and so
>> does seat 3.
>>
>> In the many-gamepads-per-seat model, the question rises again of where you
>> put the gamepads. Do they go in the same seat as the touchpad they're
>> attached to? Or do they all end up in seat 1? If they're all in seat 1, does
>> the player indicator on their controller match up with their actual player
>> number? And again, they all have to individually focus the game.
>
> Again, how does the focus get changed? Where are the gamepads deriving
> their focus from? Put them in that seat, and add a wl_touch object for
> every gamepad.

But each seat only has one wl_touch object, no? So you *need* a separate 
seat for each player. We would *like* all the wl_gamepad/wl_touch pairs 
to derive their focus from seat 1, but there's no way to do that without 
focus slaving.

>>> If the focus can be changed independently, how does this happen?
>>
>> Focus can only be changed independently for "collaborative" seats.
>> "Collaborative mode" means "I want to interact with the desktop
>> independently." When collaborative mode is turned off, your focus is derived
>> from a seat which *is* in collaborative mode.
>
> OK, so what's the downside from having the gamepad embedded inside the
> seat it derives its focus from?

The downside happens whenever players are using an input device other 
than a gamepad. In those scenarios, you're requiring every client to 
roll their own solution for assigning player devices. I would rather the 
compositor handle device assignment.

This isn't without precedent. All three major consoles allow you to 
press or hold the home button and choose an option to reassign 
controllers, and that's built into the OS rather than the game.

You're also rendering the player ID on game controllers unusable. The 
only way to make the player LEDs correct in these scenarios is to shunt 
the work off to the client, but that's not without cost to the server 
either. You would need to add a new request to the gamepad object to set 
the state of your player indicator, and the server would have to 
remember the state of each gamepad for each client.

>>> If
>>> the keyboard/pointer seat switches focus to another game, do both
>>> seats switch, or does the other stay behind?
>>
>> Assuming seat 2 is following seat 1, they both switch.
>>
>>> If both switch - why are
>>> we complicating the focus model rather than just adding both to a
>>> seat?
>>
>> Because this problem doesn't start and end with gamepads. If we take the
>> multiple-gamepads-per-seat model to its logical conclusion, then to support
>> the scenarios above, you would also need multiple mice/keyboards/touchpads
>> per seat. And then how do you determine which devices belong to which players?
>
> No, we don't because we don't lose information by aggregating them.
> If you want to support multiple independent mice, then the expectation
> is that the mice can keep on going straight out of the window, and
> then change the focus.  So either it changes the focus for everyone
> (same seat), or for just it (separate seat).
>
> If you want relative mice, Kristian's relative work should probably be
> extended to provide one object per device, similar to wl_gamepad,
> since aggregating it becomes a lot less useful.

What about keyboards? Let's say you have two players, each with their 
own keyboard, both trying to move with WASD. Multiple seats is the only 
way to accomplish this, unless we also provide a way to lock multiple 
keyboards per seat?

> But if it doesn't have any involvement or interaction with the core
> shell UI, then why is it a seat? A seat has a pretty singular meaning
> right now, and I'd rather not complicate and change that for all
> clients, and make them contort themselves in order to support a fairly
> niche usecase.

This is why I mentioned Lemmings, Settlers, and Hired Guns. These 
scenarios aren't inherently niche, they've just been rendered niche by 
OS design choices.

> Maybe I'm just being thick, but I honestly can't see the downsides of
> the multiple-gamepads-per-seat model.  One seat is currently
> restricted to one of each device, but that's only because we were able
> to aggregate those devices without losing information.  For gamepads,
> that's blatantly not true, but they don't have interactions with the
> shell and core UI - that's driven by the other devices in the seat, or
> by the shell's own policy - so the usual answer of 'put them in
> another seat' doesn't apply I think.

I think our misunderstanding is because you're picturing a traditional 
console, where gamepads are the one and only way to interact with a 
game. Indeed, in that scenario, there's no downsides to the 
many-gamepads-per-seat model. But does that model make sense when both 
players expect to be able to use a mouse/keyboard combo? Or when one 
player wants to use mouse/keyboard, and another wants to use a gamepad?

Once you concede that it makes more sense to put the player ID in the 
seat rather than the gamepad, it no longer makes sense to put multiple 
gamepads in each seat.

-Rick-