[PATCH weston 21/25] protocol: add weston_touch_calibration

[Peter Hutterer]

On Fri, Mar 23, 2018 at 02:01:01PM +0200, Pekka Paalanen wrote:
> From: Pekka Paalanen <pekka.paalanen at collabora.co.uk>
> 
> This is a Wayland protocol extension to allow the calibration of
> touchscreens in Weston.
> 
> See: https://phabricator.freedesktop.org/T7868
> 
> Signed-off-by: Pekka Paalanen <pekka.paalanen at collabora.co.uk>
> ---
>  Makefile.am                           |   5 +-
>  protocol/weston-touch-calibration.xml | 320 ++++++++++++++++++++++++++++++++++
>  2 files changed, 324 insertions(+), 1 deletion(-)
>  create mode 100644 protocol/weston-touch-calibration.xml
> 
> diff --git a/Makefile.am b/Makefile.am
> index e028a2a1..5e830777 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -168,7 +168,9 @@ nodist_libweston_ at LIBWESTON_MAJOR@_la_SOURCES =				\
>  	protocol/pointer-constraints-unstable-v1-protocol.c		\
>  	protocol/pointer-constraints-unstable-v1-server-protocol.h      \
>  	protocol/input-timestamps-unstable-v1-protocol.c		\
> -	protocol/input-timestamps-unstable-v1-server-protocol.h
> +	protocol/input-timestamps-unstable-v1-server-protocol.h		\
> +	protocol/weston-touch-calibration-protocol.c			\
> +	protocol/weston-touch-calibration-server-protocol.h
>  
>  BUILT_SOURCES += $(nodist_libweston_ at LIBWESTON_MAJOR@_la_SOURCES)
>  
> @@ -1534,6 +1536,7 @@ EXTRA_DIST +=					\
>  	protocol/weston-screenshooter.xml	\
>  	protocol/text-cursor-position.xml	\
>  	protocol/weston-test.xml		\
> +	protocol/weston-touch-calibration.xml	\
>  	protocol/ivi-application.xml		\
>  	protocol/ivi-hmi-controller.xml
>  
> diff --git a/protocol/weston-touch-calibration.xml b/protocol/weston-touch-calibration.xml
> new file mode 100644
> index 00000000..b1e19f9b
> --- /dev/null
> +++ b/protocol/weston-touch-calibration.xml
> @@ -0,0 +1,320 @@
> +<?xml version="1.0" encoding="UTF-8"?>
> +<protocol name="weston_touch_calibration">
> +
> +  <copyright>
> +    Copyright © 2017 Collabora, Ltd.
> +    Copyright © 2017 General Electric Company
> +
> +    Permission is hereby granted, free of charge, to any person obtaining a
> +    copy of this software and associated documentation files (the "Software"),
> +    to deal in the Software without restriction, including without limitation
> +    the rights to use, copy, modify, merge, publish, distribute, sublicense,
> +    and/or sell copies of the Software, and to permit persons to whom the
> +    Software is furnished to do so, subject to the following conditions:
> +
> +    The above copyright notice and this permission notice (including the next
> +    paragraph) shall be included in all copies or substantial portions of the
> +    Software.
> +
> +    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> +    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> +    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
> +    THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> +    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
> +    FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
> +    DEALINGS IN THE SOFTWARE.
> +  </copyright>
> +
> +  <interface name="weston_touch_calibration" version="1">
> +    <description summary="weston touchscreen calibration interface">
> +      This is the global interface for calibrating a touchscreen input
> +      coordinate transformation. It is recommended to make this interface
> +      privileged.
> +
> +      This interface can be used by a client to show a calibration pattern and
> +      receive uncalibrated touch coordinates, facilitating the computation of
> +      a calibration transformation that will align actual touch positions
> +      on screen with their expected coordinates.
> +
> +      Immediately after being bound by a client, the server sends the
> +      touch_device events.

s/server/compositor/, in a few more places

> +
> +      The client chooses a touch device from the touch_device events,
> +      creates a wl_surface and then a weston_touch_calibrator for the
> +      wl_surface and the chosen touch device. The client waits for the server
> +      to send a configure event before it starts drawing the first calibration
> +      pattern. After receiving the configure event, the client will iterate
> +      drawing a pattern, getting touch input via weston_touch_calibrator,
> +      and converting pixel coordinates to expected touch coordinates with
> +      weston_touch_calibrator.convert until it has enough correspondences to
> +      compute the calibration transformation or the server cancels the
> +      calibration.
> +
> +      Once the client has successfully computed a new calibration, it can use
> +      weston_touch_calibration.save request to load the new calibration into
> +      the server. The server may take this new calibration into use and may
> +      write it into persistent storage.
> +    </description>
> +
> +    <enum name="error">
> +      <description summary="global interface errors"/>
> +      <entry name="invalid_surface" value="0"
> +             summary="the given wl_surface already has a role"/>
> +      <entry name="invalid_device" value="1"
> +             summary="the given device is not valid"/>
> +      <entry name="already_exists" value="2"
> +             summary="a calibrator has already been created"/>
> +    </enum>
> +
> +    <request name="destroy" type="destructor">
> +      <description summary="unbind">
> +	Destroy the binding to the global interface, does not affect any
> +	objects already created through this interface.
> +      </description>
> +    </request>
> +
> +    <request name="create_calibrator">
> +      <description summary="give the calibrator role to a surface">
> +	This gives the calibrator role to the surface and ties it with the
> +	given touch input device.
> +
> +	The surface must not already have a role, otherwise invalid_surface
> +	error is raised.
> +
> +	The device string must be one advertised with touch_device event's
> +	device argument, otherwise invalid_device error is raised.
> +
> +	There must not exist a weston_touch_calibrator protocol object in the
> +	server already, otherwise already_exists error is raised. This
> +	limitation is server-wide and not specific to any particular client.

Personal preference: I find it easier to understand when a sentence is "if
blah then error E". As opposed to "if not blah, otherwise E". It also
narrows down the error cases a bit better too because you're only describing
the error case rather than the not-error case.

> +      </description>
> +      <arg name="surface" type="object" interface="wl_surface"
> +           summary="the surface to give the role to"/>
> +      <arg name="device" type="string" summary="the touch device to calibrate"/>
> +      <arg name="cal" type="new_id" interface="weston_touch_calibrator"
> +           summary="a new calibrator object"/>
> +    </request>
> +
> +    <request name="save">
> +      <description summary="save calibration for a touch device">
> +	This request asks the server to save the calibration data for the
> +	given touch input device. The server may ignore this request.
> +
> +	The device string must be one advertised with touch_device event's
> +	device argument, otherwise invalid_device error is raised.
> +
> +	The array must contain exactly six 'float' (the 32-bit floating
> +	point format used by the C language on the system) numbers. The numbers
> +	form a libinput style 2-by-3 calibration matrix in row-major order.

'libinput-style', i.e. hyphenated. but tbh, no need to mention libinput
here, just spell out the matrix:
"""
 For a 3D matrix in the form
   ( a b c )
   ( d e f )
   ( 0 0 1 )
 the array must contain the values [a, b, c, d, e, f] with a at index 0.
"""

Much easier than figuring out what "row-major order" means.
   

> +      </description>
> +      <arg name="device" type="string" summary="the target touch device"/>
> +      <arg name="matrix" type="array" summary="the new calibration matrix"/>
> +    </request>
> +
> +    <event name="touch_device">
> +      <description summary="advertise a touchscreen input device">
> +	When a client binds to weston_touch_calibration, one touch_device
> +	event is sent for each touchscreen that is available to be calibrated.
> +	These events are not sent later even if new touch devices appear.

Unclear what "later" means, use "Touch devices added after the initial burst
of events will not generate a touch_device event".

Though arguably - why not? Because it makes things easier to implement?

> +
> +	An event carries the touch device identification and the associated
> +	output or head (display connector) name.
> +
> +	On platforms using udev, the device identification is the udev DEVPATH.

do we want to really set this in stone in the protocol? Also, the DEVPATH
is without /sys, right? Why not make this an open format and let the client
figure it out. It's not hard to strcmp for /sys or /dev/input/event in the
client and iirc we have precedence for that in the tablet protocol. or in
libinput for tablets, can't remember now :)

> +      </description>
> +      <arg name="device" type="string"
> +           summary="the touch device identification"/>
> +      <arg name="head" type="string"
> +           summary="name of the head or display connector"/>
> +    </event>
> +  </interface>
> +
> +  <interface name="weston_touch_calibrator" version="1">
> +    <description summary="calibrator surface for a specific touch device">
> +      On creation, this object is tied to a specific touch device. The
> +      server sends a configure event which the client must obey with the
> +      associated wl_surface.
> +
> +      Once the client has committed content to the surface, the server can
> +      grab the touch input device, prevent it from emitting normal touch events,
> +      show the surface on the correct output, and relay input events from the
> +      touch device via this protocol object.
> +
> +      Touch events from other touch devices than the one tied to this object
> +      must generate wrong_touch events on at least touch-down and must not
> +      generate normal or calibration touch events.
> +
> +      At any time, the server can choose to cancel the calibration procedure by
> +      sending the cancel_calibration event. This should also be used if the
> +      touch device disappears or anything else prevents the calibration from
> +      continuing on the server side.
> +
> +      If the wl_surface is destroyed, the server must cancel the calibration.
> +
> +      The touch event coordinates and conversion results are delivered in
> +      calibration units. Calibration units are in the closed interval
> +      [0.0, 1.0] mapped into 32-bit unsigned integers. An integer can be

should probably add what 0.0 and 1.0 represent on each axis, i.e. nominal
width and height of the device's sensor. Or is this something we need to
hide?

> +      converted into a real value by dividing by 2^32-1. A calibration matrix
> +      must be computed from the [0.0, 1.0] real values, but the matrix elements
> +      do not need to fall into that range.
> +    </description>
> +
> +    <enum name="error">
> +      <description summary="calibrator object errors"/>
> +      <entry name="bad_size" value="0"
> +             summary="surface size does not match"/>
> +      <entry name="not_mapped" value="0"
> +             summary="requested operation is not possible without mapping the surface"/>
> +    </enum>
> +
> +    <request name="destroy" type="destructor">
> +      <description summary="destroy the calibrator">
> +	This unmaps the surface if it was mapped. The input device grab
> +	is dropped, if it was present. The surface loses its role as a
> +	calibrator.
> +      </description>
> +    </request>
> +
> +    <request name="convert">
> +      <description summary="convert from surface to raw coordinates">
> +	This request asks the server to convert the surface-local coordinates
> +	into the expected touch input coordinates appropriate for the
> +	associated touch device.
> +
> +	The coordinates given as arguments to this request are relative to
> +	the associated wl_surface.
> +
> +	If a client asks for conversion before it has committed valid
> +	content to the wl_surface, the not_mapped error is raised.
> +
> +	If the server has cancelled the calibration, the conversion result
> +	shall be zeroes.
> +
> +	The intention is that a client uses this request to convert marker
> +	positions that the user is supposed to touch during calibration.
> +      </description>
> +      <arg name="x" type="int" summary="surface-local X coordinate"/>
> +      <arg name="y" type="int" summary="surface-local Y coordinate"/>
> +      <arg name="reply" type="new_id" interface="weston_touch_coordinate"
> +           summary="object delivering the result"/>
> +    </request>

in theory, you could use this to figure out where on the screen you are
which is something we've avoided elsewhere. Since it's very confined I'm not
sure you could do anything with that information but is this something we
need to worry about?

> +
> +    <event name="configure">
> +      <description summary="surface size">
> +	This event tells the client what size to make the surface. The client
> +	must obey the size exactly on the next commit with a wl_buffer.
> +
> +	This event shall be sent once as a response to creating a
> +	weston_touch_calibrator object.
> +      </description>
> +      <arg name="width" type="int" summary="surface width"/>
> +      <arg name="height" type="int" summary="surface height"/>
> +    </event>
> +
> +    <event name="cancel_calibration">
> +      <description summary="cancel the calibration procedure">
> +	This is sent when the server wants to cancel the calibration and
> +	drop the touch device grab. The server unmaps the surface, if
> +	it was mapped.
> +
> +	The weston_touch_calibrator object will not send any more events. The
> +	client should destroy it.
> +      </description>
> +    </event>
> +
> +    <event name="wrong_touch">

"invalid_device_touch" maybe? 'wrong' feels.... wrong :)
also, while explained in the description, wrong_touch could also refer to a
touch that's outside the boundary, at the wrong time, etc. All these things
one could assume if one doesn't read the docs.

Cheers,
   Peter

> +      <description summary="user touched a wrong touchscreen">
> +	Calibration can only be done on one touch input device at a time. If
> +	the user touches another touch device during calibration, the server
> +	sends this event to notify the calibration client. The client should
> +	show feedback to the user that he touched a wrong screen. This is
> +	useful particularly when it is impossible for a user to tell which
> +	screen he should touch (when the screens are cloned).
> +      </description>
> +    </event>
> +
> +    <!-- touch events copied from wl_touch interface -->
> +    <event name="down">
> +      <description summary="touch down event and beginning of a touch sequence">
> +	A new touch point has appeared on the surface. This touch point is
> +	assigned a unique ID. Future events from this touch point reference
> +	this ID. The ID ceases to be valid after a touch up event and may be
> +	reused in the future.
> +
> +	For the coordinate units, see weston_touch_calibrator.
> +      </description>
> +      <arg name="time" type="uint" summary="timestamp with millisecond granularity"/>
> +      <arg name="id" type="int" summary="the unique ID of this touch point"/>
> +      <arg name="x" type="uint" summary="x coordinate in calibration units"/>
> +      <arg name="y" type="uint" summary="y coordinate in calibration units"/>
> +    </event>
> +
> +    <event name="up">
> +      <description summary="end of a touch event sequence">
> +	The touch point has disappeared. No further events will be sent for
> +	this touch point and the touch point's ID is released and may be
> +	reused in a future touch down event.
> +      </description>
> +      <arg name="time" type="uint" summary="timestamp with millisecond granularity"/>
> +      <arg name="id" type="int" summary="the unique ID of this touch point"/>
> +    </event>
> +
> +    <event name="motion">
> +      <description summary="update of touch point coordinates">
> +	A touch point has changed coordinates.
> +
> +	For the coordinate units, see weston_touch_calibrator.
> +      </description>
> +      <arg name="time" type="uint" summary="timestamp with millisecond granularity"/>
> +      <arg name="id" type="int" summary="the unique ID of this touch point"/>
> +      <arg name="x" type="uint" summary="x coordinate in calibration units"/>
> +      <arg name="y" type="uint" summary="y coordinate in calibration units"/>
> +    </event>
> +
> +    <event name="frame">
> +      <description summary="end of touch frame event">
> +	Indicates the end of a set of events that logically belong together.
> +	A client is expected to accumulate the data in all events within the
> +	frame before proceeding.
> +
> +	A wl_touch.frame terminates at least one event but otherwise no
> +	guarantee is provided about the set of events within a frame. A client
> +	must assume that any state not updated in a frame is unchanged from the
> +	previously known state.
> +      </description>
> +    </event>
> +
> +    <event name="cancel">
> +      <description summary="touch session cancelled">
> +	Sent if the compositor decides the touch stream is a global
> +	gesture. No further events are sent to the clients from that
> +	particular gesture. Touch cancellation applies to all touch points
> +	currently active on this client's surface. The client is
> +	responsible for finalizing the touch points, future touch points on
> +	this surface may reuse the touch point ID.
> +      </description>
> +    </event>
> +    <!-- END of touch events copied from wl_touch interface -->
> +
> +  </interface>
> +
> +  <interface name="weston_touch_coordinate" version="1">
> +    <description summary="coordinate conversion reply"/>
> +
> +    <!-- no destructor request defined, no requests possible -->
> +
> +    <event name="result">
> +      <description summary="coordinates in raw touch space">
> +	This event returns the conversion result from surface coordinates to
> +	raw touch device coordinates.
> +
> +	For details, see weston_touch_calibrator.convert. For the coordinate
> +	units, see weston_touch_calibrator.
> +
> +	This event destroys the weston_touch_coordinate object.
> +      </description>
> +      <arg name="x" type="uint" summary="x coordinate in calibration units"/>
> +      <arg name="y" type="uint" summary="y coordinate in calibration units"/>
> +    </event>
> +  </interface>
> +</protocol>
> -- 
> 2.16.1
> 
> _______________________________________________
> wayland-devel mailing list
> wayland-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/wayland-devel
>