Input and games.

[Pekka Paalanen]

On Fri, 26 Apr 2013 14:28:30 -0400
Todd Showalter <todd at electronjump.com> wrote:

> On Thu, Apr 25, 2013 at 8:50 AM, Pekka Paalanen <ppaalanen at gmail.com>
> wrote:
> 
> > Todd has already listed what features a standard gamepad or
> > controller has. Now someone should start designing the protocol. :-)
> 
>     Based on the wl_pointer docs, I should think it would look
> something like this:
> 
> ----------------------------------8<----------------------------------
> 
> wl_gamepad::connect -- gamepad appears
>     The connect event occurs when a gamepad is plugged in, attaches
> via wireless, or otherwise becomes available for use.  The message can
> also be generated in response to a client "enumerate gamepads" query.
> 
>     The name argument may or may not be empty, depending on what
> information the server has available, but if present can be used by
> the client to determine what to do with wl_gamepad::extended events.
> 
>     The pad_index is used to tell messages from different gamepads
> apart; a system may have multiple gamepads connected, and messages
> from them need to be distinguishable.  The pad_index value may be much
> higher than the number of gamepads currently connected if the user has
> been connecting and disconnecting gamepads in pathological ways.
> 
>     The cookie may or may not contain useful data, and shouldn't be
> counted on, but it is a hint to the application using the data whether
> a connecting gamepad has been seen before.  If the hardware has a
> software-visible serial number, the cookie should be a hash of that
> value.  If the hardware is plugging in at a specific USB port, the
> cookie should be a hash of the device path.  The cookie exists so that
> if the application sees a gamepad disappear and then another gamepad
> appears, if the cookie for the old and new controllers match it can
> assume it has the same physical gamepad.
> 
> Arguments:
>     time -- uint -- standard event timestamp
>     name -- string -- device name
>     pad_index -- uint -- which gamepad this is
>     cookie -- uint -- unique device hash; UNRELIABLE, hint only
> 
> ----------------------------------------------------------------
> wl_gamepad::disconnect -- gamepad disappears
>     The disconnect event occurs when a gamepad becomes unavailable,
> either due to unplugging or signal loss.
> 
> Arguments:
>     time -- uint -- standard event timestamp
>     pad_index -- uint -- which gamepad this is
> 

Hi Todd,

a problem here is that to receive a wl_gamepad::connect event, you
first have to create a wl_gamepad protocol object, which is a bit
counterintuitive.

A wl_gamepad protocol object should correspond to a single physical
device. So, we would have a wl_gamepad object per each contoller, and
you do not need the pad_index argument in any of the events. This
should be easier on the game programmer, too, since you can attach
user data to each protocol object in libwayland-client, and use that in
the callbacks without any global data.

That leaves the question, where do we put the connect and disconnect
event equivalents. Below you mention also keyboards, so instead of
playing ad-hoc, let's use wl_seats the way they are designed.

A wl_seat would need to grow a capability bit for wl_gamepad, and a
request to create a wl_gamepad object. When you use that request to
create a wl_gamepad, the first thing it does is send its description:
name and cookie in your proposal, as an extra event type.

That limits us to one wl_gamepad device per wl_seat, so a server needs
to create more wl_seats for more controllers. That shouldn't be any
problem, these seats would only have the gamepad capability by default.

If your gamepad actually had a keyboard, or maybe even a touchpad (if
that is supposed to be used as a pointer device), it would simply be
advertised as a standard wl_keyboard or wl_pointer on the wl_seat. Each
player would have their own wl_seat, and it is obvious which keyboard
belongs with which gamepad.

If we or the user wants the system keyboard and mouse, i.e. the things
not on a gamepad, as a separate seat instead of merged with one of the
gamepads, that is simply a server configuration thing. Think about a
desktop GUI, where you tick a box "keep system keyboard separate", kind
of thing.

Oh, and the disconnect event. The standard wl_seat way for that seems
to be a new capabilities event, with the gamepad bit unset.


All this still leaves some details unsolved, like which Wayland client
should receive the gamepad events on a wl_seat? The one having the
keyboard focus? Probably. But what if a wl_seat has no wl_keyboard or
wl_pointer? Just the first client that creates a wl_gamepad? What if
you have two games running, and you want to switch between them? Should
a wl_gamepad have its own focused surface attribute? How do you assign
that focus? If you have other input devices in addition to a gamepad on
a wl_seat, how do you get all their foci to the game, when the user
wants it? How does the user indicate he wants it?

Many of these questions are tied also to the shell protocol, since the
shell may have requests for asking or rejecting some foci. (We don't
have that yet in desktop-shell, but it has been mentioned.)

> ----------------------------------------------------------------
> wl_gamepad::stick -- gamepad stick movement
>     A stick event occurs when there is stick movement on a gamepad.
> It would be preferable if the protocol could handle float data, but
> failing that the axis values can be mapped to a large integer range,
> as below.  The precision of the fixed type is sufficient for most
> current hardware, but not all; for example, the ps3 controller analog
> sticks have a [-128 .. 127] range, but the ps3 dpad is actually
> pressure sensitive, and therefore actually has an effective range of
> [-255 .. 255].  It's not hard to imagine higher-precision controllers
> in the future as prices come down.
> 
>     The stick_index indicates which stick the message pertains to; for
> hardware with more than the standard number of joysticks/thumbsticks,
> higher index values are possible, but 0 is always left stick, 1 is
> always right stick and 2 is always the dpad.  Even if the physical
> hardware lacks one or more of those axis values, additional axis
> values will be mapped above 2.
> 
> Arguments:
>     time -- uint -- standard event timestamp
>     pad_index -- uint -- which gamepad this is
>     stick_index -- uint -- 0 for left stick, 1 for right stick, 2 for
> dpad x -- int -- the x axis of the stick mapped to [-2^15 .. 2^15 - 1]
>     y -- int -- the y axis of the stick mapped to [-2^15 .. 2^15 - 1]

All int and uint are 32-bit in the protocol, btw, should be enough
precision for intervals like [0, 1] and [-1, 1], I think.

I agree that the fixed type is not really suitable here. It was
designed to hold pixel coordinates foremost.

> ----------------------------------------------------------------
> wl_gamepad::trigger -- gamepad analog trigger movement
>     A trigger event occurs when there is analog trigger movement on a
> gamepad.  As with stick messages, it would be preferable if the axis
> value could be sent as float, but failing that the value can be mapped
> to a large integer range.
> 
>     The trigger_index is 0 for left stick values and 1 for right stick
> values.  Hardware with more triggers can potentially supply higher
> values; the pressure-sensitive buttons on the ps3 controller would go
> here, for instance.
> 
> Arguments:
>     time -- uint -- standard event timestamp
>     pad_index -- uint -- which gamepad this is
>     trigger_index -- uint -- 0 for left trigger, 1 for right trigger
>     x -- uint -- the trigger value mapped to [0 .. 2^15 - 1]
> 
> ----------------------------------------------------------------
> wl_gamepad::button -- gamepad button press
>     A button event occurs when a button is pressed or released.  The
> standard buttons are:
> 
> 0 - BUTTON_FACE_NORTH -- triangle on ps3, y on xbox
> 1 - BUTTON_FACE_EAST -- circle on ps3, b on xbox
> 2 - BUTTON_FACE_SOUTH -- x on ps3, a on xbox
> 3 - BUTTON_FACE_WEST -- square on ps3, x on xbox
> 4 - BUTTON_SHOULDER_LEFT -- L1 on ps3, LT on xbox
> 5 - BUTTON_SHOULDER_RIGHT -- R1 on ps3, RT on xbox
> 6 - BUTTON_LEFT_STICK -- left stick click
> 7 - BUTTON_RIGHT_STICK -- right stick click
> 8 - BUTTON_START -- start button
> 
>     Controllers may have other buttons, and if so they must map to
> index values higher than those of the standard buttons.  Nonstandard
> buttons can only be understood in the context of the information
> delivered via the wl_gamepad::connect event.
> 
>     There is perhaps a question here about whether to deal with things
> like controller keyboards; some game controllers have keyboards that
> connect to them for things like in-game messaging.  Arguably they
> belong within the gamepad protocol if they're going to be handled,
> since they're per-player keyboards.  That said, they're also uncommon.
>  If they are going to be handled, it also makes sense to ask whether
> this is actually something where we should be bringing in the wl_seat
> abstraction, but that might be abusing wl_seat.  The alternative would
> be to do something like use keyboard keysyms, but set the high bit.
> Regardless, I'm not sold on including them in the protocol at all.
> Call it an open question.
> 
> Arguments:
>     time -- uint -- standard event timestamp
>     pad_index -- uint -- which gamepad this is
>     button_index -- uint -- index of the button that changed state
>     state -- uint -- 0 for released, 1 for pressed
> 
> ----------------------------------------------------------------
> wl_gamepad::accelerometer
>     An optional part of the protocol; an orientation event indicates a
> change in accelerometer data.  Accelerometer data is assumed to be
> generated as a three axis vector; some hardware apparently produces
> quaternions, which is interesting, but quaternions (at least,
> normalized quaternions) don't give you velocity, just orientation.
> Besides, without float values in the protocol the data the quaternion
> encoding gets icky fast; quaternions are normalized 4d vectors, which
> means they need a lot of precision below the decimal.
> 
>     This is a part of the protocol that is being included for
> discussion; I'm not sold on it.
> 
> Arguments:
>     time -- uint -- standard event timestamp
>     pad_index -- uint -- which gamepad this is
>     accel_index -- uint -- index of the accelerometer that changed
> state x -- uint -- accelerometer x axis, mapped such that 1.0f ==
> 2^15 - 1 y -- uint -- accelerometer y axis, mapped such that 1.0f ==
> 2^15 - 1 z -- uint -- accelerometer z axis, mapped such that 1.0f ==
> 2^15 - 1
> 
> ----------------------------------------------------------------
> wl_gamepad::sysbutton -- gamepad system button event
>     A sysbutton event occurs when the system button (the ps button on
> a ps3 controller, the glowy x button on the xbox 360 controller, the
> home button on the wii controller) is pressed.  While this information
> might be passed on to the application, it is somewhat expected that
> his event will be trapped and acted upon by the window manager.
> 
> Arguments:
>     time -- uint -- standard event timestamp
>     pad_index -- uint -- which gamepad this is

I don't think we need a separate event for this, just the normal button
event is enough. If the display server wants to eat the event, it can
do so in any case. Or was there some other reason for this?

> ----------------------------------------------------------------
> wl_gamepad::extended -- gamepad hardware-specific extended event
>     This is an optional extension to the protocol; a method of
> handling extra data created by gamepads beyond the standard protocol.
> Most extended information would pass through more standard messages;
> extra buttons, sticks or trigger values should use those messages with
> higher index values.  This message is for handling anything that
> doesn't fit the standard model.  I'm not sold on this event either; it
> might just be a bad idea.
> 
> Arguments:
>     time -- uint -- standard event timestamp
>     pad_index -- uint -- which gamepad this is
>     subtype -- uint -- ordinal identifying the event subtype
>     a -- uint -- first parameter
>     b -- uint -- second parameter
>     c -- uint -- third parameter
>     d -- uint -- fourth parameter

I agree with Jason here, the interface can be extended later as needed,
so no need for a generic catch-all event.


Thanks,
pq