[PATCH weston 1/3] Introduce pointer lock interface

[Jonas Ådahl]

On Wed, Sep 24, 2014 at 06:04:26PM +0200, Matthieu Gautier wrote:
> Le 24/09/2014 16:41, Jason Ekstrand a écrit :
> >On Wed, Sep 24, 2014 at 1:46 AM, Matthieu Gautier <dev at mgautier.fr
> ><mailto:dev at mgautier.fr>> wrote:
> >
> >    Hi Jason,
> >
> >    Well, I'll try to be more concise.
> >
> >    In fact, there are two unrelated parts in my proposition :
> >
> >    # First part : getting relative events
> >
> >    Passing through a wl_pointer that may not exists to get relative
> >    events seems a bit odd to me.
> >
> >    My main idea is to have a new object (I've call wl_relative) *not
> >    related* to wl_pointer that send to client the relative events.
> >    If a client want relative events, it open this object and receive
> >    events from it. Simply.
> >    No need to look the wl_pointer or anything else.
> >
> >    The wl_pointer still continue to work as usual. There is strictly
> >    no change in that part.
> >
> >    The only relation we may find between wl_relative and wl_pointer
> >    is that relative events are sent only when the wl_surface as the
> >    focus.
> >    In a first approximation, we can say that the wl_relative focus is
> >    the same of wl_pointer focus and that relative events are send
> >    only when wl_pointer.motion are send (ie, between wl_pointer.enter
> >    and wl_pointer.leave)
> >    But a special compositor could follow its own rules.
> >
> >
> >What is a relative input device?  How does the client know if/how it's
> >related to the wl_pointer?
> >
> >The first question is the hard one.  A pointer is a relatively well-dfined
> >thing.  We've had mice for ~30 years now and everyone knows what to expect
> >from one. They give you motion (sort-of relative), they have buttons (at
> >least one, but usually at least 3), they frequently have at least one axis
> >of relative motion that we call "scroll".  A mouse is a well-defined input
> >device and we know what to do with one.  We also know that, classically,
> >they move this little cursor around on the screen in a way that's defined
> >by the window system and is usually in absolute coordinates as far as
> >applications are concerned.
> >
> >What is a relative input device?  It gives relative motion.  How many axes
> >does it have?  How should each axis be interpreted by applications?  What
> >kind of pseudo-physical model should we use to understand it.  (a mouse
> >cursor "moves" on the screen).  Does it have buttons?  How many?  What do
> >they do?  In order for the protocol to be useful for applications, all of
> >those questions need well-defined answers.  If it's just "gives relative
> >events" then we're right back to the horrific "remap your joystick for
> >every game" problem we had in the '90s only now you have to remap your
> >mouse too.  In fact, one of the things about mice that's great for game
> >devs is precisely that they always know what they're getting.
> >
> >Along those lines, there was a protocol proposed for gamepads some time
> >ago.  (you could go dig it up on the list and read about it).  The basic
> >idea there was that most gamepad designs have more-or-less coalesced into
> >the XBox/PS form-factor of two sticks, a d-pad, 4 buttons, 2 or 4
> >triggers, and a "start" buton.  You can go online and find dozens of
> >different companies shipping gamepads that all have this same basic
> >layout.  The idea was to then create a protocol based on that layout so
> >that game developers could have a common base to build on.  The reason I
> >bring this up is that the entire reason why the protocol was feasible at
> >all is that the concept of a gamepad has become reasonably well-defined to
> >the point where you could write a common protocol that works for most
> >combinations of game and gamepad.
> 
> I have to agree with you. Trying to write the xml corresponding to my idea
> point out this problem.
> 
> Not cause of the definition of the wl_relative (I planned to just have
> motion events) but cause without buttons wl_relative is mostly useless
> without wl_pointer.
> And it doesn't fulfill the constraint for system with a relative device but
> without a pointer interpretation of this device.
> 
> However, split the problem in two parts is still relevant for me. We don't
> need a lock on the pointer to get the relative movement of the mouse.
> 
> Why not just add new arguments to the wl_pointer.motion event and drop the
> idea of a wl_relative object ? [*]
> Those new arguments (delta_x and delta_y) corresponding to the untransformed
> relative movement.
> 
> The pointer lock part being unchanged.
> 
> If for any reason the user move the mouse and the pointer position doesn't
> change (cause of pointer lock or pointer is already on a screen corner),
> a motion event is still generated but with surface_x/surface_y being the
> same as the previous ones.
> 
> [*] I'm not totally aware of what is the general consensus on how to extend
> wayland protocol.
> Maybe we try to avoid change in the core protocol and want to make change
> through extension. In this case, indeed, we cannot just add arguments to the
> motion event.

Hi,

I'm just gonna jump in to the discussion in the middle here. It sounds
like the proposal regarding wl_relative is to generalize a set of input
devices into a category that produce "relative" motion events. To me
this sounds fairly vague and it seems not to be a well known kind of
device that fits in its own protocol object. Without actually knowing
exactly what device is generating the relative motions, its hard to make
any sense of them. It sounds more like you want some way to bypass that
the compositor abstracts the input devices for you so you can process
your special device; for this I think we more likely need a "raw input
device" protocol more close to evdev, assuming we don't want to just let
clients open any input device itself (which would enable key loggers
etc).

Another thing is that with a wl_relative is that we would emit events
from one or many devices via multiple protocol objects at the same time,
but at the same time wl_pointer would represent one set of devices,
while wl_relative another (partially overlapping) set.

For example, if we have a tablet device which is used as a pointer
device, there are no relative events to send as the tablet produces
absolute events (absolute pointer events even, AFAIU). Another device,
one that you seem to describe, would only send relative events, as it is
not a pointer device. Yet another device, a mouse, would send out events
on both wl_pointer and wl_relative. How could the client make any sense
of these? For the wl_pointer case, its pretty clear that the client
should simply interpret them as pointer events, but for the wl_relative,
it more seems that the client needs to really know what device generates
the events. Assuming the wl_relative is like wl_pointer, wl_keyboard and
wl_touch, i.e. multiplexes multiple devices making them act as one
virtual device, it makes it even harder to interpret what wl_relative
actually means, as it is no longer possible to assume it comes from the
one special device.

IMHO, we should create protocol abstractions for input device types that
are well defined, such as pointer devices, keyboards, touch devices,
drawing tablets and possibly (xbox/ps3) like gamepads. But for special
use case devices that are not possible to abstract and put behind a
commonly understood device type, I think it would work better with some
way for clients to process raw events itself.


Jonas

> 
> >
> >As a small note, a classic mouse isn't really relative because the
> >position on screen sort-of corresponds to the position on the mouse pad.
> >Sure, it's sloppy and subject to acceleration, but it's not truly relative
> >like a trackball.  Not that this really matters, but it's worth pointing
> >out.
> I'm still thinking that tho pointer stuff is artificial construction based
> on relative events. This construction is of course extremely useful and well
> integrated by user.
> One can imagine a UI without pointer and with relative motion (FPS game is a
> example, they use relative events but don't have a notion of pointer)
> 
> But, I think this point is no more relevant in this discussion. Having
> wayland supports this kind of future potential interface would probably need
> to many change for a usage that may not worth it.
> 
> >
> >    # Second part : pointer lock
> >
> >    The pointer lock should not be related to relative events. This is
> >    two different parts.
> >
> >    To acquire a pointer lock, a client just ask it.
> >    It can be by the request wl_pointer.get_lock(lock_type)    (I've
> >    called it set_mode in my past mail but get_lock is a better name)
> >
> >    This request return a wl_callback used by the compositor to end
> >    the lock (either cause it refuse it or the lock normally ends)
> >    The lock can be released by the client using
> >    wl_pointer.release_lock() (former reset_mode)
> >
> >    The lock_type can be :
> >    - lock : Do not move the pointer (What was in the Jonas' proposal)
> >    - confine : Confine the pointer into the surface.
> >
> >    During a lock, all events are still send as usual :
> >    - Relative events from wl_relative are not impacted by the
> >    wl_pointer lock
> >    - wl_pointer.motion are still send in confine mode
> >    - wl_pointer button, axis events are still sent
> >
> >    # Remarks and QA:
> >
> >    The main difference compared to Jonas' proposal is that I totally
> >    separate the relative events problem from the pointer lock
> >    problem. (I maybe should have start by this :) )
> >    In Jonas' proposal you have to acquire the pointer lock to get
> >    relative events. In my proposition we can get one without having
> >    the other.
> >
> >    I don't especially care if this is my proposal or Jonas' ones that
> >    is kept. But separate the two part in unrelated problems seems
> >    better to me.
> >
> >    I've you give me times, I will send you a patch of wayland.xml to
> >    describe more precisely what I'm thinking about.
> >    It could be interesting that Jonas update its protocol xml with
> >    remarks from others to be sure that we compare up to date proposal
> >    and not miss anything.
> >
> >    - Why not let the application open the device ?
> >     => For security reasons. We probably don't want that any
> >    applications can open input devices and read from them all events.
> >     => Scalability. Devices are not always mouses. Applications may
> >    not know how to read them.
> >                           In my past job, I've made a android
> >    application that connect to the STB in Wifi and discuss directly
> >    with the compositor to send input events. There was no device to
> >    read from except a socket.
> >    - What my proposal can do that Jonas' one can't do ?
> >     => The only thing I can find is the case where a seat doesn't
> >    have a wl_pointer but have relative events. For all the rest,
> >    Jonas' proposal seems to work.
> >     => My proposition seems simpler to me. Less dependencies between
> >    elements, less questions about who create and release objects.
> >(But It's maybe also cause it's mine :) )
> >
> >    Regards,
> >    Matthieu.
> >
> >    Le 23/09/2014 23:42, Jason Ekstrand a écrit :
> >>    Hi Matthieu,
> >>    Could you please provide more explanation on what use-cases you
> >>    are considering and why the current proposal fails to accomplish
> >>    them?  All I was able to get from your post was the example of a
> >>    TV remote.  It's all well and good to completely disagree with
> >>    the proposed approach, but without a good reason as to why it
> >>    doesn't work, it's kind of moot.
> >>
> >>    It's also worth noting the scope of the proposal.  The primary
> >>    objective here is to provide a mechanism to take a device that
> >>    could be providing relative events but which the compositor has
> >>    turned into a pointer and get the relative events out of it. It's
> >>    not intended to be a generic system for getting relative events.
> >>
> >>    On that note, if you want to get a generic relative motion
> >>    events, can't you just open the device and read them?  Sure we
> >>    could cobble together a specification for how to read a bunch of
> >>    buttons and relative events and then create a cursor from them.
> >>I'm pretty sure it exists in the form of the USB HID spec.  How
> >>    does reimplementing that in Wayland help us? There was, at one
> >>    point, a proposal for gamepads, but nothing has happened there in
> >>    a while.
> >>
> >>    On Tue, Sep 23, 2014 at 9:01 AM, Matthieu Gautier
> >>    <dev at mgautier.fr <mailto:dev at mgautier.fr>> wrote:
> >>
> >>        Hello,
> >>
> >>        I'm pretty new into wayland and the discussion is relatively
> >>        long, so I may have missed arguments/constraints.
> >>        However I would like to share my point of view.
> >>
> >>        It seems to me that we are taking the problem the wrong way.
> >>        Relative motions exist as soon as there is a device
> >>        generating them. wl_pointer is just a particular
> >>        interpretation of those events.
> >>
> >>        In fact, we may have a system where we have relative motion
> >>        events but no wl_pointer. Think about a smart tv with a
> >>        remote control with accelerator/gyroscope detectors.
> >>        This remote may behave as a mouse, generating relative motion
> >>        events. But the main interface of the TV may have no pointer.
> >>        The interface should been a set of icons and user move
> >>        between them with the remote buttons.
> >>
> >>
> >>    If you're just pressing buttons to move between icons, then
> >>    "pointer" is probably the wrong way to look at it anyway.  It's
> >>    more "arrow keys" than a pointer.
> >    Exactly, there is no pointer in this example. But there are
> >    relative events applications may want to have.
> >
> >>        In the same way, we may want that special applications still
> >>        have access to motion events:
> >>        - A web browser that will display itself the pointer (or
> >>        activate wl_pointer in the compositor)
> >>        - A video game
> >>        - Any application that want make gesture recognition.
> >>
> >>        In this context, wl_pointer is a special use-case of a shell
> >>        and having a mouse device doesn't imply having a pointer.
> >>
> >>        Relative motions should be always available (if there is a
> >>        device) and wl_pointer should be created on top of relative
> >>        motions.
> >>        Trying to reduce the wl_pointer behavior to have the raw
> >>        events seems to me the contrary of what we have to do.
> >>
> >>
> >>        What I propose is :
> >>
> >>        - Having a way to get "relative input object" (lets call it
> >>        wl_relative for now) from wl_seat.
> >>        - Having a way to get a wl_pointer from the wl_seat at it is
> >>        already the case.
> >>
> >>        Relative events a sent to client if it is active (It is to
> >>        the compositor to decide this, as usual) whatever there is a
> >>        wl_pointer or not.
> >>
> >>        The pointer lock interface will become some kind of
> >>        "deactivate/configure wl_pointer".
> >>
> >>        # Functionally :
> >>
> >>        A combination of :
> >>        - Hide the cursor (already available with wl_pointer.set_cursor)
> >>        - Don't not update wl_pointer position from relative events.
> >>        - Confine the pointer position into my wl_surface.
> >>        - Set wl_pointer at this position.
> >>
> >>        - A fps game will hide the cursor and deactivate update of
> >>        wl_pointer position.
> >>        - A strategy game will just confine the pointer.
> >>        - A application with a 3D view that what to rotate it when
> >>        user drag the mouse will just deactivate update of pointer
> >>        position between button_down and button_up.
> >>        - A application that just want relative motion events do nothing.
> >>
> >>        At any time, relative motion events are sent to client
> >>        through the wl_relative object. Regardless of the state of
> >>        wl_pointer.
> >>        It is up to the client to handle events from wl_pointer or
> >>        wl_relative depending of which kind of information it wants.
> >>
> >>
> >>        # Interface :
> >>
> >>        The wl_pointer could gain two (four?) more requests :
> >>
> >>        - set_mode(mode, callback)
> >>        - reset_mode()
> >>        ( - has_mode
> >>        - get_mode )
> >>
> >>        The default mode is the mode we have for now (no special
> >>        constraints)
> >>        A client can change the mode of a wl_pointer. It gets a callback.
> >>        When compositor stops the special mode (or refuse it) the
> >>        done event of the callback is sent.
> >>        When the client has finished with special mode, it sends the
> >>        reset_mode request.
> >>
> >>        The wl_pointer.leave event may or not be sent to client when
> >>        the done event is sent (The pointer may still being inside
> >>        the wl_surface when special mode ends)
> >>        However a wl_pointer.leave event implies a done event. (We
> >>        cannot have a special mode if we don't have the pointer focus)
> >>
> >>
> >>        On the interface to get the wl_relative object from seat, it
> >>        depends :
> >>
> >>        Is there a possibility to have several cursor on one seat ?
> >>
> >>
> >>    One cursor per seat.  If there are multiple physical mice, then
> >>    you get an aggregate of all the mice.  You can, however, have
> >>    multiple seats in which case you will get multiple pointers.
> >>
> >>        (Two mouses moving two cursors)
> >>        Does wl_seat.get_cursor return always a proxy to the same
> >>        object ?
> >>
> >>
> >>    Yes, all pointers created from the same seat are identical.
> >>
> >>
> >>        - If there is only one cursor, we can simply add a
> >>        get_relative request to wl_seat.
> >>        - If not, we should get a object from the other.
> >>          . Ideally, get the wl_pointer from wl_relative. (and
> >>        wl_relative from wl_seat)
> >>          . Practically, cause of the existent, get wl_relative from
> >>        wl_pointer.
> >>
> >>
> >>        Regards,
> >>        Matthieu Gautier.
> >>
> >>        _______________________________________________
> >>        wayland-devel mailing list
> >>        wayland-devel at lists.freedesktop.org
> >>        <mailto:wayland-devel at lists.freedesktop.org>
> >>        http://lists.freedesktop.org/mailman/listinfo/wayland-devel
> >>
> >>
> >>
> >>
> >>    _______________________________________________
> >>    wayland-devel mailing list
> >>    wayland-devel at lists.freedesktop.org  <mailto:wayland-devel at lists.freedesktop.org>
> >>    http://lists.freedesktop.org/mailman/listinfo/wayland-devel
> >
> >
> >    _______________________________________________
> >    wayland-devel mailing list
> >    wayland-devel at lists.freedesktop.org
> >    <mailto:wayland-devel at lists.freedesktop.org>
> >    http://lists.freedesktop.org/mailman/listinfo/wayland-devel
> >
> >
> >
> >
> >_______________________________________________
> >wayland-devel mailing list
> >wayland-devel at lists.freedesktop.org
> >http://lists.freedesktop.org/mailman/listinfo/wayland-devel
> 

> _______________________________________________
> wayland-devel mailing list
> wayland-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/wayland-devel