EFL/Wayland and xdg-shell

[Daniel Stone]

Hi,
Replies to both here ...

On 15 April 2015 at 02:39, Carsten Haitzler <raster at rasterman.com> wrote:
> On Tue, 14 Apr 2015 01:31:56 +0100 Daniel Stone <daniel at fooishbar.org> said:
>> On 14 April 2015 at 01:02, Bryce Harrington <bryce at osg.samsung.com> wrote:
>> > While window rotation was used more as an example of how built-in
>> > assumptions in the API could unintentionally constrain D-E's, than as a
>> > seriously needed feature, they did describe a number of ideas for rather
>> > elaborate window behaviors:
>> >
>> >   * Rotation animations with frame updates to allow widget re-layouts
>> >     while the window is rotating.
>>
>> So not just animating the transition, but requesting the client
>> animate the content as well? That's extremely esoteric, and seems like
>> it belongs in a separate extension - which is possible.
>
> not esoteric - an actual request from people making products.

The reason I took that as 'esoteric' was that I assumed it was about
free window rotation inside Weston: a feature which is absolutely
pointless but as a proof-of-concept for a hidden global co-ordinate
space. Makes a lot more sense for whole-display rotation. More below.

>> >   * Non-linear surface movement/resizing animations and transition
>> >     effects.
>>
>> This seems like a compositor thing?
>
> it is.

i.e. no bearing on actual xdg_shell

>> > There was lots of interest in hearing more about Wayland's plans for
>> > text-cursor-position and input-method, which are necessary for Asian
>> > languages.
>>
>> It's sadly not been unmaintained for a while.
>>
>> > A particular question was how clients could coordinate with
>> > the virtual keyboard input window so that it doesn't overlay where text
>> > is being inserted.
>>
>> See the text_cursor_position protocol.
>
> actually the other way around... clients know where the vkbd region(s) are so
> client can shuffle content to be visible. :)

In a VKB (rather than overlay-helper, as used for complex composition)
scenario, I would expect xdg-shell to send a configure event to resize
the window and allow for the VKB. If this isn't sufficient - I
honestly don't know what the behaviour is under X11 - then a potential
version bump of wl_text could provide for this.

>> RandR is a disaster of an API to expose to clients. I would suggest
>> that anything more than a list of monitors (not outputs/connectors)
>> with their resolutions, relative monitor positioning, and the ability
>> to change/disable the above, is asking for trouble.
>
> agreed - exposing randr is not sane. it's an internal compositor matter at this
> level of detail (if compositor chooses to have a protocol, do it al itself
> internally etc. is up to it, but any tool to configure screen output at this
> level would be compositor specific).
>
> what i do think is needed is a list of screens with some kind of types attached
> and rough metadata like one screen is left or right of another (so clients like
> flight simulators could ask to have special surface on the left/right screens
> showing views out the sides of the cockpit and middle screen is out the front).
> something like:
>
> 0 desktop primary
> 1 desktop left_of primary
> 2 desktop right_of primary
> 3 mobile detached
> 4 tv above primary
>
> (pretend a phone with 4 external monitors attached).

Hell of a phone. More seriously, yes, a display-management API could
expose this, however if the aim is for clients to communicate intent
('this is a presentation') rather than for compositors to communicate
situation ('this is one of the external monitors'), then we probably
don't need this. wl_output already provides the relative geometry, so
all that is required for this is a way to communicate output type.

> perhaps listing
> resolution, rotation and dpi as well (pick the screen with the biggest physical
> size or highest dpi - adjust window contents based on screen rotation - eg so
> some controls are always facing the "bottom edge" of the screen where some
> button controls are - the screen shows the "legend" text).
>
> apps should not be configuring any of this. it's read-only.

This already exists today - wl_output's geometry (DPI, rotation,
subpixel information, position within global co-ordinate space) and
mode (mode) events. So, no problem.

> surfaces should be
> able to hint at usage - eg "i want to be on the biggest tv". "i want to be
> wherever you have a small mobile touch screen" etc. compositor deals with
> deciding where they would go based on the current state of the world
> screen-wise and app hints.

Right. So if we do have this client-intent-led interface (which would
definitely be the most Wayland-y approach), then we don't need to
advertise output types and wl_output already deals with the rest, so
no change required here?

>> > One area we could improve on X for output configuration is in how
>> > displays are selected for a given application's surface.  A suggestion
>> > was type descriptors for outputs, such as "laptop display",
>> > "television", "projector", etc. so that surfaces could express an output
>> > type affinity.  Then a movie application could request its full screen
>> > playback surface be preferentially placed on a TV-type output, while
>> > configuration tools would request being shown on a laptop-screen-type
>> > output.
>>
>> A neat idea, but a separate extension to be sure. Flipping things
>> around a bit, you might say that the application declares its type,
>> and the compositor applies smart placement depending on its type.
>
> well at least "which screen to go on" imho would be a core hint/feature for
> xdg-shell. listing screens like above would be another extension.

Again though, Wayland attempts to be descriptive, rather than
prescriptive. In line with a popup being a separate surface type
rather than 'put this exactly here, on top, and give me all input
until I tell you otherwise', this would likely be a hint about the
type of content provided, and the compositor could make a sensible
decision about where to display it. Or you could hint as to which type
of output to display it on. This could potentially spare all clients
the effort required to explicitly select an output.

On 15 April 2015 at 02:26, Carsten Haitzler <raster at rasterman.com> wrote:
> On Mon, 13 Apr 2015 17:24:26 -0700 "Jasper St. Pierre" <jstpierre at mecheye.net>
> said:
>> On Mon, Apr 13, 2015 at 5:02 PM, Bryce Harrington <bryce at osg.samsung.com>
>> > While window rotation was used more as an example of how built-in
>> > assumptions in the API could unintentionally constrain D-E's, than as a
>> > seriously needed feature, they did describe a number of ideas for rather
>> > elaborate window behaviors:
>> >
>> >   * Rotation animations with frame updates to allow widget re-layouts
>> >     while the window is rotating.
>>
>> Why does xdg-shell / Wayland impede this?
>
> ok. hmmm. this is an actual business use case (requirement) not theory. i have
> been asked for this. let us pretend now we have a "desktop ui" where some
> windows are rotated, but some are not. when a window rotates 90 degrees is must
> ALSO resize to a new size at the same time. imagine 800x480 -> 480x800 WHILE
> rotating. but the widgets within the window have to re-layout AS the rotation
> animation happens. let us also have the compositor doing the rotation (so it's
> rotating the output buffer from the app whilst it resizes from size A to size
> B). the client would need to be told the current rotation at any point during
> this animation so it can render its layout differences accordingly as well.
> either that or the rotate animation is totally client driven and it is
> informing the compositor of the current rotation needed. imagine this. :)

Makes complete sense; I've implemented that before (on the N900 -
funnily enough, also 800x480).

In the general case (i.e. what we'd want to spec for), you could only
do this with a fullscreen app. Doing it with multiple top-level
surfaces active would be an impressive juggling act. What would have
to happen in this case is:
  - client informs compositor it is capable of managing the rotation process
  - compositor triggers beginning of rotation and informs client
  - client renders a series of frames animating between initial and final states
  - last frame is produced twice: one rotated in-client (i.e. output
displaying in non-rotated co-ordinate space will show the buffer as
rotated), one in newly-rotated co-ordinate space
  - client attaches and commits pre-rotated frame (the first)
  - client immediately attaches and commits rotated-coord-buffer (the
second), tagged with information that this is the final frame and the
compositor should atomically switch its display along with displaying
this buffer

This is relatively complex, and introduces a whole lot of failure
modes: having a 'pending rotation' will be tricky to get exactly right
without proper testcases, and we're also relying on clients to not
hold our entire display pipeline hostage, so we'd need some kind of
arbitrary timeout too I guess. It is entirely doable, but the
complexity makes me think that this would best be proven either as an
external extension, or something that was prepared to live out-of-tree
for a little while whilst it was proven.

Nothing worse than introducing complex and failure-prone API that you
then have to support ~forever, along with the new and not-broken one.
xdg_shell's core status means it's not something we're going to get to
deprecate either.

>> > There was lots of interest in hearing more about Wayland's plans for
>> > text-cursor-position and input-method, which are necessary for Asian
>> > languages.  A particular question was how clients could coordinate with
>> > the virtual keyboard input window so that it doesn't overlay where text
>> > is being inserted.  Security is also a top concern here, to ensure
>> > unauthorized clients can't steal keyboard input if (when) the virtual
>> > keyboard client crashes.
>>
>> The solution GNOME takes, which is admittedly maybe too unrealistic,
>> is that IBus is our input method framework, and thus our compositor
>> has somewhat tight integration with IBus. I don't think input methods
>> need to be part of the core Wayland protocol.
>
> i would disagree. input methods should be a core part of the protocol as they
> are input. vkbd's, or cjk or anything else is a complex case of input, but
> still input. input should come from the compsitor.

I think he was talking about the compositor <-> IM engine
co-ordination (i.e. the input_method protocol) rather than the client
<-> IM engine protocol (i.e. the text protocol). The latter should
absolutely be a first-class part of the protocol, but unfortunately
no-one with the requisite knowledge and skills (definitely not me) has
had the time to work on it since it was initially developed. Hopefully
that changes.

The compositor <-> IM engine work should absolutely not be exposed to
arbitrary clients, as handing them control over all input is a
security nightmare. The compositor certainly needs control over, and
private protocol for, the session's IM engine. Maybe that private
protocol is a thing that people can standardise on, such that any
conforming IM engine can work with any conforming compositor, but it's
certainly never going to be first-class public protocol that any
client can use. I have no idea what the challenges in standardising
this are, and/or why GNOME is tied exclusively to ibus.

> in x11 we use properties to advertise to clients which area(s) of the screen
> have been obscured by the vkbd, so clients can scroll/move/re-layout content to
> ensure the entry you are typing into isnt hidden under your on-screen keyboard.
> this is needed in wayland eventually.

No disagreement, and patches certainly welcome.

>> > For screensaver inhibition, it was suggested that this be tracked
>> > per-surface, so that when the surface terminates the inhibition is
>> > removed (this is essentially what xdg-screensaver tries to do, although
>> > is specific to the client process rather than window iirc).
>>
>> It's similar to
>> http://standards.freedesktop.org/idle-inhibit-spec/latest/re01.html
>>
>> It would require careful semantics that I'm not so sure about. Why is
>> it being tied to the surface rather than the process important?
>
> because inhibition is about an app saying "my window content is important -
> keep it visible and don't blank/dim etc.". people brute-force use a global
> control because that is all they have had, but it is really about content. that
> surface is playing a movie - when doing so - tag that surface as "please don't
> blank me!" (and likely another flag like "please don't dim me!"). if the
> surface is destroyed, hidden, moved to another virtual desktop - the inhibition
> can be lifted by the compositor as the content's desire to be always visible is
> now irrelevant due to context. and no-fullscreen mode does not equate to
> "inhibit screensaver".

Yes, exactly.

>> > They also
>> > suggested per-output blanking support so e.g. laptop lvds could be
>> > blanked but the projector be inhibited, or when watching a movie on
>> > dual-head, hyst the non-movie head powersaves off.  They also suggested
>> > having a 'dim' functionality which would put the display to maximum
>> > dimness rather than blanking it completely; I'm not sure on the use case
>> > here or how easy it'd be to implement.
>>
>> This is stuff I would hope would be provided and implemented by the
>> DE. As a multimonitor user who quite often watches things on one
>> monitor while coding on another, I'd turn this feature off.
>
> thus why it should be on a surface. if my movie playback is on my external "tv"
> monitor and it asks to inhibit - the compositor will stop blanking the external
> screen, but it may happily blank the internal screen on the laptop - which is
> exactly what we really would want. if the information is per-surface, this kind
> of smart behavior is possible. there is no need for apps to know about multiple
> monitors and which to inhibit if the inhibition is tied to a surface.

Which somewhat ties into the point I made earlier about descriptive
intent ('this surface is a movie') rather than imperative commands
('don't blank wl_output at 17').

>> > I had hoped to discuss collaboration on testing, but without specifics
>> > there didn't seem to be strong interest.  One question was about
>> > collecting protocol dumps for doing stability testing or performance
>> > comparison/optimization with; while we're not doing that currently, that
>> > sounded straightforward and like it could be useful to investigate more.
>> >
>> >
>> > There was some confusion over what the purpose of xdg-shell really is;
>> > it was looked at as a reference implementation rather than as a
>> > lowest-common denominator that should be built *onto*.  So it seems that
>> > Wayland have some messaging to do to ensure xdg-shell really is
>> > 'cross-desktop'.
>>
>> xdg-shell should be seen as the proper shell protocol everybody should
>> strive to support. It's entirely possible that DEs have their own
>> protocols they put on top: GTK+ has gtk-shell, for instance.
>
> we also will likely extend too with some e-shell as it means we can prototype
> and implement new features rapidly, and then consider bringing them up for
> standardization in xdg-shell.

This is definitely the right thing to do.

Cheers,
Daniel