[RFC] protocol: Extend wayland seat with interfaces for sensor inputs.
Stefan Schmidt
s.schmidt at samsung.com
Fri Aug 16 06:17:39 PDT 2013
Hello.
Sorry for the late reply. Got sick in between.
First of all thanks for taking the time to review this and provide feedback.
On 08/12/2013 04:54 PM, David Herrmann wrote:
> On Thu, Aug 1, 2013 at 10:39 AM, Stefan Schmidt <s.schmidt at samsung.com>
> wrote:
>> Treating some specific sensors as input devices. In particular
>> adding support for wl_compass, wl_gyroscope and wl_accelerometer here.
>>
>> We have requests to start and stop sensor event receiving as well as
>> events to receive the different axis values for these sensors.
>>
>> Signed-off-by: Stefan Schmidt <s.schmidt at samsung.com>
>> ---
>> protocol/wayland.xml | 123
> ++++++++++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 123 insertions(+)
>>
>> diff --git a/protocol/wayland.xml b/protocol/wayland.xml
>> index 3cfa953..066a718 100644
>> --- a/protocol/wayland.xml
>> +++ b/protocol/wayland.xml
>> @@ -1251,6 +1251,9 @@
>> <entry name="pointer" value="1" summary="The seat has pointer
> devices"/>
>> <entry name="keyboard" value="2" summary="The seat has one or
> more keyboards"/>
>> <entry name="touch" value="4" summary="The seat has touch
> devices"/>
>> + <entry name="compass" value="8" summary="The seat has compass
> devices"/>
>> + <entry name="gyroscope" value="16" summary="The seat has
> gyroscope devices"/>
>> + <entry name="accelerometer" value="32" summary="The seat has
> accelerometer devices"/>
>
> If we add all kinds of devices here, we will very soon run out of
> space for 32bit bitfields (which I think wayland uses for "uint"). I
> don't have a solution, but we should keep this in mind.
Lets hope for now we don't have more than 26 different device types
coming up.
>> </enum>
>>
>> <event name="capabilities">
>> @@ -1297,6 +1300,39 @@
>>
>> <!-- Version 2 of additions -->
>>
>> + <request name="get_compass" since="2">
>> + <description summary="return compass object">
>> + The ID provided will be initialized to the wl_compass interface
>> + for this seat.
>> +
>> + This request only takes effect if the seat has the compass
>> + capability.
>> + </description>
>> + <arg name="id" type="new_id" interface="wl_compass"/>
>> + </request>
>> +
>> + <request name="get_gyroscope" since="2">
>> + <description summary="return gyroscope object">
>> + The ID provided will be initialized to the wl_gyroscope
> interface
>> + for this seat.
>> +
>> + This request only takes effect if the seat has the gyroscope
>> + capability.
>> + </description>
>> + <arg name="id" type="new_id" interface="wl_gyroscope"/>
>> + </request>
>> +
>> + <request name="get_accelerometer" since="2">
>> + <description summary="return accelerometer object">
>> + The ID provided will be initialized to the wl_accelerometer
> interface
>> + for this seat.
>> +
>> + This request only takes effect if the seat has the
> accelerometer
>> + capability.
>> + </description>
>> + <arg name="id" type="new_id" interface="wl_accelerometer"/>
>> + </request>
>> +
>
> This interface limits each seat to one device of each. For keyboards
> or mouse we internally merge these into a single virtual device, but
> we cannot do this for accelerometers or gyros. We had a discussion
> about that recently, I am not sure about the conclusion, though. If
> there was one, please include it in the commit message.
You answered that yourself below. We should allow to cope with more then
one sensor device per type. I really want to avoid doing that right now
though. Having the protocol to allow for it but not doing it right now.
> Besides, why are these devices bound to seats? Where are these devices
> supposed to be? An accelerometer can be attached to an input device,
> an HDD, a laptop, a tablet, ... If you look at this from a
> smartphone/tablet view, it is obvious. But for desktops it really
> isn't clear whose data an accelerometer device reports.
Indeed phones and tablets are my main focus right now but it should work
for desktops and laptops as well. It should be a sensor that is really
meant for input though. An accelerometer inside your harddisk might be
used for that (played games with it on an older Thinkpad) but is not the
use case here. Neither are background apps or services.
I would like to reduce this proposal to real input for apps and games.
The wayland protocol should not define sensor access for the whole system.
> I guess these devices are meant to be attached to the input device
> that this seat/user uses. In this case it makes sense to allow only a
> single device and bind them to seats. But please clarify this in the
> description. No one cares if the description is 1000 words long, so
> feel free to be verbose.
Next version will be more verbose. Means another chance to spot what I
missed. :)
>> <event name="name" since="2">
>> <description summary="unique identifier for this seat">
>> In a multiseat configuration this can be used by the client to
> help
>> @@ -1605,6 +1641,93 @@
>> </event>
>> </interface>
>>
>> + <interface name="wl_compass" version="2">
>> + <description summary="compass sensor device">
>> + The wl_compass interface represents a compass
>> + associated with a seat.
>> + </description>
>> +
>> + <request name="start">
>> + <description summary="Start receiving events">
>> + Sent to enable event receiving for the compass sensor.
>> + </description>
>> + </request>
>> +
>> + <request name="stop">
>> + <description summary="Stop receiving events">
>> + Sent to disable event receiving for the compass sensor.
>> + </description>
>> + </request>
>
> So this is a shared device. Please note that down. A compositor has to
> broadcast events to all interested clients, otherwise this API doesn't
> work (or is inconsistent). Did you think of the implications of that?
> Might be fine, but I don't see any discussion, so please elaborate.
> What are the reasons to allow background applications to access
> compass/accelerometer/gyro? That is, why didn't you follow the
> enter/leave API?
>
> I don't want change your API, but the short descriptions give me the
> impression that you didn't think about these implications so I'd like
> to see some explanations. And the longer the descriptions, the less
> bikeshed you get.. as usual.
I thought about them in the beginning but did not took them into account
when actually writing the protocol changes. So you have a point here. :)
Should we have a sensor grab here where the currently focused app has
the custody over the incoming sensor data? What about different apps
running at the same time? Like real multitasking.
I would be glad to get some input on this. The proposal can be reworked
its not in stone.
> This actually poses a security risk, please have a look at:
>
> http://arstechnica.com/apple/2011/10/researchers-can-keylog-your-pc-using-
> your-iphones-accelerometer/
Yeah, I already know about it.
>> +
>> + <event name="motion">
>> + <description summary="Motion event coming from the compass
> sensor">
>> + Updated sensor data available from compass
>> + </description>
>> + <arg name="x" type="fixed" summary="x-axis"/>
>> + <arg name="y" type="fixed" summary="y-axis"/>
>> + <arg name="z" type="fixed" summary="z-axis"/>
>> + </event>
>
> I never worked with compass devices on linux and it isn't obvious to
> me what these 3 axes represent? An explanation would really help here.
> But maybe I am just missing some context..
Added more explanation to it. Looks like this right now and will be in
the next patch:
<event name="motion">
<description summary="Motion event coming from the compass sensor">
Updated sensor data available from compass
Measurement of the ambient magnetic field in the X, Y and Z axis.
Coordinate system (device in portrait mode with screen facing
upwards):
X axis goes from the left to the right side of the device
Y axis goes from the bottom of the screen to the top
Z axis goes from the back of the screen to the front of the screen
</description>
<arg name="x" type="fixed" summary="x-axis in micro Tesla (uT)"/>
<arg name="y" type="fixed" summary="y-axis in micro Tesla (uT)"/>
<arg name="z" type="fixed" summary="z-axis in micro Tesla (uT)"/>
>> +
>> + </interface>
>> +
>> + <interface name="wl_gyroscope" version="2">
>> + <description summary="gyroscope sensor device">
>> + The wl_gyroscope interface represents a gyroscope
>> + associated with a seat.
>> + </description>
>> +
>> + <request name="start">
>> + <description summary="Start receiving events">
>> + Sent to enable event receiving for the gyroscope sensor.
>> + </description>
>> + </request>
>> +
>> + <request name="stop">
>> + <description summary="Stop receiving events">
>> + Sent to disable event receiving for the gyroscope sensor.
>> + </description>
>> + </request>
>> +
>> + <event name="motion">
>> + <description summary="Motion event coming from the gyroscope
> sensor">
>> + Updated sensor data available from gyroscope
>> + </description>
>> + <arg name="x" type="fixed" summary="x-axis"/>
>> + <arg name="y" type="fixed" summary="y-axis"/>
>> + <arg name="z" type="fixed" summary="z-axis"/>
>> + </event>
>
> Here it is obvious what the 3 axes mean, but we should clarify that
> these are randomly chosen. At least I am not aware of any definitions
> which axes x, y and z are (for instance regarding the yaw/pitch/roll
> definitions).
Updated version looks like this:
<event name="motion">
<description summary="Motion event coming from the gyroscope sensor">
Updated sensor data available from gyroscope
Measurement of the rate of rotation around X, Y and Z axis in
degree per second.
Values are observed in positive notation in the
counter-clockwise direction.
Coordinate system (device in portrait mode with screen facing
upwards):
X axis goes from the left to the right side of the device (Pitch)
Y axis goes from the bottom of the screen to the top (Roll)
Z axis goes from the back of the screen to the front of the
screen (Azimuth/Yaw)
</description>
<arg name="x" type="fixed" summary="x-axis in degree per second"/>
<arg name="y" type="fixed" summary="y-axis in degree per second"/>
<arg name="z" type="fixed" summary="z-axis in degree per second"/>
</event>
>> +
>> + </interface>
>> +
>> + <interface name="wl_accelerometer" version="2">
>> + <description summary="accelerometer sensor device">
>> + The wl_accelerometer interface represents a accelerometer
>> + associated with a seat.
>> + </description>
>> +
>> + <request name="start">
>> + <description summary="Start receiving events">
>> + Sent to enable event receiving for the accelerometer sensor.
>> + </description>
>> + </request>
>> +
>> + <request name="stop">
>> + <description summary="Stop receiving events">
>> + Sent to disable event receiving for the accelerometer sensor.
>> + </description>
>> + </request>
>> +
>> + <event name="motion">
>> + <description summary="Motion event coming from the accelerometer
> sensor">
>> + Updated sensor data available from accelerometer
>> + </description>
>> + <arg name="x" type="fixed" summary="x-axis"/>
>> + <arg name="y" type="fixed" summary="y-axis"/>
>> + <arg name="z" type="fixed" summary="z-axis"/>
>> + </event>
>
> Again, I think these are randomly chosen, right?
And this looks as f <event name="motion">
<description summary="Motion event coming from the accelerometer
sensor">
Updated sensor data available from accelerometer
Measurement of the acceleration of G on the axis in m/s^2.
Coordinate system (device in portrait mode with screen facing
upwards):
X axis goes from the left to the right side of the device
Y axis goes from the bottom of the screen to the top
Z axis goes from the back of the screen to the front of the screen
</description>
<arg name="x" type="fixed" summary="x-axis G in m/s^2"/>
<arg name="y" type="fixed" summary="y-axis G in m/s^2"/>
<arg name="z" type="fixed" summary="z-axis G in m/s^2"/>
</event>ollowing now:
> (copied from your reply)
>> We started with many more sensors but brought it down to these three
>> as they seem to make the most sense. I kept the proposal small
>> regarding exposed requests and events to not over-engineer it from the
> beginning.
>
> Would you mind listing some more devices? Just to get a feeling what
> this discussion was all about.
Modern to of the line phones have tons of sensors nowadays. Some a real
hardware sensor and some are just virtual ones aggregated from
different sources.
The list I have here would be like this:
SENSOR_ACCELEROMETER, /**< Accelerometer */
SENSOR_GRAVITY, /**< Gravity sensor */
SENSOR_LINEAR_ACCELERATION, /**< Linear
acceleration sensor */
SENSOR_DEVICE_ORIENTATION, /**< Device
orientation sensor */
SENSOR_MAGNETIC, /**< Magnetic sensor */
SENSOR_ORIENTATION, /**< Orientation sensor */
SENSOR_GYROSCOPE, /**< Gyroscope sensor */
SENSOR_LIGHT, /**< Light sensor */
SENSOR_PROXIMITY, /**< Proximity sensor */
SENSOR_MOTION_SNAP, /**< Snap motion sensor */
SENSOR_MOTION_SHAKE, /**< Shake motion
sensor */
SENSOR_MOTION_DOUBLETAP, /**< Double tap motion
sensor */
SENSOR_MOTION_PANNING, /**< Panning motion
sensor */
SENSOR_MOTION_PANNING_BROWSE, /**< Panning browse
motion sensor */
SENSOR_MOTION_TILT, /**< Tilt motion sensor */
SENSOR_MOTION_FACEDOWN, /**< Face to down
motion sensor */
SENSOR_MOTION_DIRECTCALL, /**< Direct call
motion sensor */
SENSOR_MOTION_SMART_ALERT, /**< Smart alert
motion sensor */
SENSOR_MOTION_NO_MOVE, /**< No move motion
sensor */
SENSOR_PEDOMETER_STEP, /**< Pedometer stop
count */
> I like this interface a lot. As I see it, it addresses the needs of
> user-input. That is, it's targeted at games or apps for orientation.
> It's not targeted at background daemons to protect your hard-drive.
Correct.
> Wasn't obvious to me right from the beginning, but maybe it should
> have been? A short comment wouldn't hurt, anyway.
Will add something about it.
> That said, it makes a lot of sense to provide this data attached to a
> seat. We see a seat as a way to identify a user interacting with a
> system. So we only stuff devices in a single seat that a user can use
> in parallel. We identify the user this way. So it's immediately clear
> what an compass attached to a seat reports. The orientation of the
> user.
> However, this API is strictly bound to smartphone/tablet input. For
> instance what happens if the user has one input device in the left and
> one in the right hand? Both might report movement data, but we
> couldn't report it via this API because it assumes a single input
> device. And this isn't even some fancy look into the future.With
> Wii-Remotes we already have such input which can be greatly used. Just
> thinking about it... But if we design this API in a way that multiple
> devices can be attached to a seat but explicitly limit it to 1 device
> in this revision, then we can extend this API in the future if we want
> to support such setups. We might even tack your feet.. or head.. or..
> ok, stop me!
I would like to cover the only sensor per type per seat for now. And yes
it should be done in a way where it is extensible for more sensors.
> All in all nice to see this being worked on. Thanks!
Always good to hear that. :)
I will send out another version of this next week.
regards
Stefan Schmidt
More information about the wayland-devel
mailing list