EFL/Wayland and xdg-shell

[Carsten Haitzler (The Rasterman)]

On Wed, 15 Apr 2015 20:29:32 +0100 Daniel Stone <daniel at fooishbar.org> said:

> Hi,
> Replies to both here ...
> 
> On 15 April 2015 at 02:39, Carsten Haitzler <raster at rasterman.com> wrote:
> > On Tue, 14 Apr 2015 01:31:56 +0100 Daniel Stone <daniel at fooishbar.org> said:
> >> On 14 April 2015 at 01:02, Bryce Harrington <bryce at osg.samsung.com> wrote:
> >> > While window rotation was used more as an example of how built-in
> >> > assumptions in the API could unintentionally constrain D-E's, than as a
> >> > seriously needed feature, they did describe a number of ideas for rather
> >> > elaborate window behaviors:
> >> >
> >> >   * Rotation animations with frame updates to allow widget re-layouts
> >> >     while the window is rotating.
> >>
> >> So not just animating the transition, but requesting the client
> >> animate the content as well? That's extremely esoteric, and seems like
> >> it belongs in a separate extension - which is possible.
> >
> > not esoteric - an actual request from people making products.
> 
> The reason I took that as 'esoteric' was that I assumed it was about
> free window rotation inside Weston: a feature which is absolutely
> pointless but as a proof-of-concept for a hidden global co-ordinate
> space. Makes a lot more sense for whole-display rotation. More below.

not just whole display - but now imagine a table with a screen and touch and 4
people around it one along each side and multiple windows floating about like
scraps of paper... just an illustration where you'd want window-by-window
rotation done by compositor as well.

> >> > There was lots of interest in hearing more about Wayland's plans for
> >> > text-cursor-position and input-method, which are necessary for Asian
> >> > languages.
> >>
> >> It's sadly not been unmaintained for a while.
> >>
> >> > A particular question was how clients could coordinate with
> >> > the virtual keyboard input window so that it doesn't overlay where text
> >> > is being inserted.
> >>
> >> See the text_cursor_position protocol.
> >
> > actually the other way around... clients know where the vkbd region(s) are
> > so client can shuffle content to be visible. :)
> 
> In a VKB (rather than overlay-helper, as used for complex composition)
> scenario, I would expect xdg-shell to send a configure event to resize
> the window and allow for the VKB. If this isn't sufficient - I
> honestly don't know what the behaviour is under X11 - then a potential
> version bump of wl_text could provide for this.

no - resizing is a poorer solution. tried that in x11. first obvious port of
call. imagine vkbd is partly translucent... you want it to still be over window
content. imagine a kbd split onto left and right halves, one in the middle of
the left and right edges of the screen (because screen is bigger). :)

> >> RandR is a disaster of an API to expose to clients. I would suggest
> >> that anything more than a list of monitors (not outputs/connectors)
> >> with their resolutions, relative monitor positioning, and the ability
> >> to change/disable the above, is asking for trouble.
> >
> > agreed - exposing randr is not sane. it's an internal compositor matter at
> > this level of detail (if compositor chooses to have a protocol, do it al
> > itself internally etc. is up to it, but any tool to configure screen output
> > at this level would be compositor specific).
> >
> > what i do think is needed is a list of screens with some kind of types
> > attached and rough metadata like one screen is left or right of another (so
> > clients like flight simulators could ask to have special surface on the
> > left/right screens showing views out the sides of the cockpit and middle
> > screen is out the front). something like:
> >
> > 0 desktop primary
> > 1 desktop left_of primary
> > 2 desktop right_of primary
> > 3 mobile detached
> > 4 tv above primary
> >
> > (pretend a phone with 4 external monitors attached).
> 
> Hell of a phone. More seriously, yes, a display-management API could
> expose this, however if the aim is for clients to communicate intent
> ('this is a presentation') rather than for compositors to communicate
> situation ('this is one of the external monitors'), then we probably
> don't need this. wl_output already provides the relative geometry, so
> all that is required for this is a way to communicate output type.

i was thinking a simplified geometry. then again client toolkits can figure
that out and present a simplified enum or what not to the app too. but yes -
some enumerated "type" attached to the output would be very nice. smarter
clients can decide their intent based on what is listed as available - adapt to
the situation. dumber ones will just ask  for a fixed type and "deal with it"
if they don't get it.

> > perhaps listing
> > resolution, rotation and dpi as well (pick the screen with the biggest
> > physical size or highest dpi - adjust window contents based on screen
> > rotation - eg so some controls are always facing the "bottom edge" of the
> > screen where some button controls are - the screen shows the "legend" text).
> >
> > apps should not be configuring any of this. it's read-only.
> 
> This already exists today - wl_output's geometry (DPI, rotation,
> subpixel information, position within global co-ordinate space) and
> mode (mode) events. So, no problem.
> 
> > surfaces should be
> > able to hint at usage - eg "i want to be on the biggest tv". "i want to be
> > wherever you have a small mobile touch screen" etc. compositor deals with
> > deciding where they would go based on the current state of the world
> > screen-wise and app hints.
> 
> Right. So if we do have this client-intent-led interface (which would
> definitely be the most Wayland-y approach), then we don't need to
> advertise output types and wl_output already deals with the rest, so
> no change required here?

well the problem here is the client is not aware of the current situation. is
that output on the right a tv on the other side of the room, ore a projector, or
perhaps an internal lcd panel? is it far from the user or touchable (touch
surface). if it's touchable the app may alter ui (make buttons bigger - remove
scrollbars to go into a touch ui mode as opposed ro mouse driven...). maybe app
is written for multitouch controls specifically and thus a display far from the
user with a single mouse only will make the app "useless"? app should be able
to know what TYPE of display it is on - what types are around and be able to
ask for a type (may or may not get it). important thing is introducing the
concept of a type and attaching it to outputs (and hints on surfaces).

> >> > One area we could improve on X for output configuration is in how
> >> > displays are selected for a given application's surface.  A suggestion
> >> > was type descriptors for outputs, such as "laptop display",
> >> > "television", "projector", etc. so that surfaces could express an output
> >> > type affinity.  Then a movie application could request its full screen
> >> > playback surface be preferentially placed on a TV-type output, while
> >> > configuration tools would request being shown on a laptop-screen-type
> >> > output.
> >>
> >> A neat idea, but a separate extension to be sure. Flipping things
> >> around a bit, you might say that the application declares its type,
> >> and the compositor applies smart placement depending on its type.
> >
> > well at least "which screen to go on" imho would be a core hint/feature for
> > xdg-shell. listing screens like above would be another extension.
> 
> Again though, Wayland attempts to be descriptive, rather than
> prescriptive. In line with a popup being a separate surface type
> rather than 'put this exactly here, on top, and give me all input
> until I tell you otherwise', this would likely be a hint about the
> type of content provided, and the compositor could make a sensible
> decision about where to display it. Or you could hint as to which type
> of output to display it on. This could potentially spare all clients
> the effort required to explicitly select an output.

sure - i agree. but letting the app know the current state of the world
type-wise would be smart.

> On 15 April 2015 at 02:26, Carsten Haitzler <raster at rasterman.com> wrote:
> > On Mon, 13 Apr 2015 17:24:26 -0700 "Jasper St. Pierre"
> > <jstpierre at mecheye.net> said:
> >> On Mon, Apr 13, 2015 at 5:02 PM, Bryce Harrington <bryce at osg.samsung.com>
> >> > While window rotation was used more as an example of how built-in
> >> > assumptions in the API could unintentionally constrain D-E's, than as a
> >> > seriously needed feature, they did describe a number of ideas for rather
> >> > elaborate window behaviors:
> >> >
> >> >   * Rotation animations with frame updates to allow widget re-layouts
> >> >     while the window is rotating.
> >>
> >> Why does xdg-shell / Wayland impede this?
> >
> > ok. hmmm. this is an actual business use case (requirement) not theory. i
> > have been asked for this. let us pretend now we have a "desktop ui" where
> > some windows are rotated, but some are not. when a window rotates 90
> > degrees is must ALSO resize to a new size at the same time. imagine 800x480
> > -> 480x800 WHILE rotating. but the widgets within the window have to
> > re-layout AS the rotation animation happens. let us also have the
> > compositor doing the rotation (so it's rotating the output buffer from the
> > app whilst it resizes from size A to size B). the client would need to be
> > told the current rotation at any point during this animation so it can
> > render its layout differences accordingly as well. either that or the
> > rotate animation is totally client driven and it is informing the
> > compositor of the current rotation needed. imagine this. :)
> 
> Makes complete sense; I've implemented that before (on the N900 -
> funnily enough, also 800x480).
> 
> In the general case (i.e. what we'd want to spec for), you could only
> do this with a fullscreen app. Doing it with multiple top-level
> surfaces active would be an impressive juggling act. What would have
> to happen in this case is:

i wouldn't see this as a problem. could be done relatively easily with multiple
surfaces at the same time. :)

>   - client informs compositor it is capable of managing the rotation process
>   - compositor triggers beginning of rotation and informs client
>   - client renders a series of frames animating between initial and final
> states
>   - last frame is produced twice: one rotated in-client (i.e. output
> displaying in non-rotated co-ordinate space will show the buffer as
> rotated), one in newly-rotated co-ordinate space

ahh for the fullscreen case :) i'm thinking multiple windows at once - so
compositor would rotate the surface output at composite time given an angle
the client gives it. client would resize buffer each time with the "anchor"
being at window center (need to define that actually somehow). client now just
loops from 0 to 90 degrees, with a newly sized buffer each time and newly
rendered content. you're definitely thinking the fullscreen case subset here
where you are indeed right. :)

>   - client attaches and commits pre-rotated frame (the first)
>   - client immediately attaches and commits rotated-coord-buffer (the
> second), tagged with information that this is the final frame and the
> compositor should atomically switch its display along with displaying
> this buffer
> 
> This is relatively complex, and introduces a whole lot of failure
> modes: having a 'pending rotation' will be tricky to get exactly right
> without proper testcases, and we're also relying on clients to not
> hold our entire display pipeline hostage, so we'd need some kind of
> arbitrary timeout too I guess. It is entirely doable, but the
> complexity makes me think that this would best be proven either as an
> external extension, or something that was prepared to live out-of-tree
> for a little while whilst it was proven.

oh sure. we'd end up doing this kind of thing regardless - in xdg and wayland
core or not. if not - we'd do an extn and have it all working etc. - then maybe
come back for "hey - this stuff works - see? how about we standardize and all
agree on the same core protocol for this now?"

> Nothing worse than introducing complex and failure-prone API that you
> then have to support ~forever, along with the new and not-broken one.
> xdg_shell's core status means it's not something we're going to get to
> deprecate either.
> 
> >> > There was lots of interest in hearing more about Wayland's plans for
> >> > text-cursor-position and input-method, which are necessary for Asian
> >> > languages.  A particular question was how clients could coordinate with
> >> > the virtual keyboard input window so that it doesn't overlay where text
> >> > is being inserted.  Security is also a top concern here, to ensure
> >> > unauthorized clients can't steal keyboard input if (when) the virtual
> >> > keyboard client crashes.
> >>
> >> The solution GNOME takes, which is admittedly maybe too unrealistic,
> >> is that IBus is our input method framework, and thus our compositor
> >> has somewhat tight integration with IBus. I don't think input methods
> >> need to be part of the core Wayland protocol.
> >
> > i would disagree. input methods should be a core part of the protocol as
> > they are input. vkbd's, or cjk or anything else is a complex case of input,
> > but still input. input should come from the compsitor.
> 
> I think he was talking about the compositor <-> IM engine
> co-ordination (i.e. the input_method protocol) rather than the client
> <-> IM engine protocol (i.e. the text protocol). The latter should
> absolutely be a first-class part of the protocol, but unfortunately
> no-one with the requisite knowledge and skills (definitely not me) has
> had the time to work on it since it was initially developed. Hopefully
> that changes.

oh ok - compositor <-> IM is a different thing (personally i think this is
custom to compositor + IM and thus he's then right - i agree totally - hell
compositors may create their own IMs themseleves rather than deal with an
external IM - but they are not required to use wayland protocol to talk to IM,
just to clients).

> The compositor <-> IM engine work should absolutely not be exposed to
> arbitrary clients, as handing them control over all input is a
> security nightmare. The compositor certainly needs control over, and
> private protocol for, the session's IM engine. Maybe that private
> protocol is a thing that people can standardise on, such that any
> conforming IM engine can work with any conforming compositor, but it's
> certainly never going to be first-class public protocol that any
> client can use. I have no idea what the challenges in standardising
> this are, and/or why GNOME is tied exclusively to ibus.

in this case - i agree. i thought he was meaning client <-> IM :) my mistake :)

> > in x11 we use properties to advertise to clients which area(s) of the screen
> > have been obscured by the vkbd, so clients can scroll/move/re-layout
> > content to ensure the entry you are typing into isnt hidden under your
> > on-screen keyboard. this is needed in wayland eventually.
> 
> No disagreement, and patches certainly welcome.
> 
> >> > For screensaver inhibition, it was suggested that this be tracked
> >> > per-surface, so that when the surface terminates the inhibition is
> >> > removed (this is essentially what xdg-screensaver tries to do, although
> >> > is specific to the client process rather than window iirc).
> >>
> >> It's similar to
> >> http://standards.freedesktop.org/idle-inhibit-spec/latest/re01.html
> >>
> >> It would require careful semantics that I'm not so sure about. Why is
> >> it being tied to the surface rather than the process important?
> >
> > because inhibition is about an app saying "my window content is important -
> > keep it visible and don't blank/dim etc.". people brute-force use a global
> > control because that is all they have had, but it is really about content.
> > that surface is playing a movie - when doing so - tag that surface as
> > "please don't blank me!" (and likely another flag like "please don't dim
> > me!"). if the surface is destroyed, hidden, moved to another virtual
> > desktop - the inhibition can be lifted by the compositor as the content's
> > desire to be always visible is now irrelevant due to context. and
> > no-fullscreen mode does not equate to "inhibit screensaver".
> 
> Yes, exactly.
> 
> >> > They also
> >> > suggested per-output blanking support so e.g. laptop lvds could be
> >> > blanked but the projector be inhibited, or when watching a movie on
> >> > dual-head, hyst the non-movie head powersaves off.  They also suggested
> >> > having a 'dim' functionality which would put the display to maximum
> >> > dimness rather than blanking it completely; I'm not sure on the use case
> >> > here or how easy it'd be to implement.
> >>
> >> This is stuff I would hope would be provided and implemented by the
> >> DE. As a multimonitor user who quite often watches things on one
> >> monitor while coding on another, I'd turn this feature off.
> >
> > thus why it should be on a surface. if my movie playback is on my external
> > "tv" monitor and it asks to inhibit - the compositor will stop blanking the
> > external screen, but it may happily blank the internal screen on the laptop
> > - which is exactly what we really would want. if the information is
> > per-surface, this kind of smart behavior is possible. there is no need for
> > apps to know about multiple monitors and which to inhibit if the inhibition
> > is tied to a surface.
> 
> Which somewhat ties into the point I made earlier about descriptive
> intent ('this surface is a movie') rather than imperative commands
> ('don't blank wl_output at 17').

agreed totally.

> >> > I had hoped to discuss collaboration on testing, but without specifics
> >> > there didn't seem to be strong interest.  One question was about
> >> > collecting protocol dumps for doing stability testing or performance
> >> > comparison/optimization with; while we're not doing that currently, that
> >> > sounded straightforward and like it could be useful to investigate more.
> >> >
> >> >
> >> > There was some confusion over what the purpose of xdg-shell really is;
> >> > it was looked at as a reference implementation rather than as a
> >> > lowest-common denominator that should be built *onto*.  So it seems that
> >> > Wayland have some messaging to do to ensure xdg-shell really is
> >> > 'cross-desktop'.
> >>
> >> xdg-shell should be seen as the proper shell protocol everybody should
> >> strive to support. It's entirely possible that DEs have their own
> >> protocols they put on top: GTK+ has gtk-shell, for instance.
> >
> > we also will likely extend too with some e-shell as it means we can
> > prototype and implement new features rapidly, and then consider bringing
> > them up for standardization in xdg-shell.
> 
> This is definitely the right thing to do.

that is the plan. once we get there, it'll be time to come back with them.

-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    raster at rasterman.com