[PATCH weston 00/17] Follow up from the last xdg-shell series

[Pekka Paalanen]

On Sun, 03 May 2015 04:21:29 +0200
Mario Kleiner <mario.kleiner.de at gmail.com> wrote:

> On 04/30/2015 03:48 PM, Pekka Paalanen wrote:
> > On Fri, 10 Apr 2015 08:01:39 +0200
> > Mario Kleiner <mario.kleiner.de at gmail.com> wrote:
> >
> >> On 04/07/2015 11:01 AM, wayland-devel-request at lists.freedesktop.org wrote:
> >>> Date: Tue,  7 Apr 2015 17:01:15 +0800
> >>> From: Jonas Ådahl <jadahl at gmail.com>
> >>> To: wayland-devel at lists.freedesktop.org
> >>> Cc: Jonas Ådahl <jadahl at gmail.com>
> >>> Subject: [PATCH weston 00/17] Follow up from the last xdg-shell series
> >>> Message-ID: <1428397292-8455-1-git-send-email-jadahl at gmail.com>
> >>> Content-Type: text/plain; charset=UTF-8
> >>>
> >>> Hi,
> >>>
> >>>
> >>> This series is a follow up mostly addressing the issues raised by Pekka
> >>> during the last series as well as some other clarifications.
> >>>
> >>> The patches them self is a better in depth description of the changes, but
> >>> there are a couple of things that I'd like to bring up more explicitly.
> >>>
> >>> 1.
> >>>
> >>> This series again bumps the unstable version and breaks backward
> >>> compatibility (it moves error enums and introduces a new one).
> >>>
> >>> If there are any more breaking changes, I think we should
> >>> push those together and avoiding unstable version 6. This brings me to
> >>> the next point:
> >>>
> >>> 2.
> >>>
> >>> Fullscreen as specified in xdg-shell is more limiting than the one in
> >>> wl_shell_surface, and it might be a good idea to consider make
> >>> fullscreening a bit more powerful in xdg_surface.
> >>>
> >>> The current difference between wl_shell_surface and xdg_surface in regard
> >>> to fullscreening is that while xdg_surface allows for specifying what
> >>> output the surface should be fullscreened on, while wl_shell_surface also
> >>> allows specifying a method (scale, driver, fill) and for the driver method
> >>> a refresh rate.
> >>>
> >>> As of this series, the method of xdg_surface.set_fullscreen is identical to
> >>> t. he wl_shell_surface 'fill' fullscreen method.
> >>>
> >>> The question is whether a 'scale' and/or a 'driver' method should be made
> >>> available.
> >>>
> >>> Such a 'scale' method would more or less overlap completely with the
> >>> wl_scaler protocol, so might be completely unnecessary. Does it provide
> >>> any benefit over using wl_scaler?
> >>>
> >>> The 'driver' method could be good thing to have, assuming it has any
> >>> benefit over scaling (i.e. specify refresh rate, (memory bandwidth?)),
> >>> as I suspect it wouldn't be any compositors policy to change monitor
> >>> resolution for any fullscreen surface where its applicable.
> >>>
> >>> So the question is, how much, if any, of the fullscreen controls should
> >>> be made available via xdg_surface's fullscreen request?
> >>>
> >>
> >> Wrt. the "driver" method and being able to request a refresh rate, yes
> >> please! I assume it would also be good for games and other performance
> >> hungry apps to avoid wasting gpu cycles / memory bandwidth on scaling
> >> etc., but my specific use case is visual presentation software for
> >> neuro-science/medical applications. If one wants to find out how eyes
> >> and brains work - how we see and perceive and recognize the visual world
> >> and also what goes wrong if we don't see that well, also how different
> >> senses work together - one needs fine control over video refresh rates,
> >> presentation timing and video output resolutions, color reproduction,
> >> exact location of pixels on the display etc., often on multi-display
> >> setups. Researchers often use precisely calibrated displays, old CRT
> >> monitors due to their variable frame rate, or specialized display
> >> equipment. For that to work we need control over the exact video mode
> >> and refresh rates, if and how a display uses dithering etc. and just
> >> rescaling images in the compositor can do rather horrible things to such
> >> controlled stimuli. Some of the research equipment parses information
> >> that is color-coded in image pixels to trigger hardware operations which
> >> are perfectly synchronized to the video stream, so a simple minimal
> >> change of pixel location, size or color due to running an unexpected
> >> video mode + rescaling and filtering could completely knock out such
> >> equipment. Application control over modesetting, at least on displays
> >> where the application is fullscreen, is essential for these desktop use
> >> cases.
> >>
> >> To port such applications from X11/GLX to Wayland, ideally i'd like to
> >> have the equivalent of as much as possible of X11's full RandR extension
> >> - global modesetting, dithering and scaling control, rotation, dynamic
> >> changes to color gamma tables etc. But unless/until such things become
> >> available, the ability to set a native video mode is a start. I just
> >> have started implementing Wayland support for my neuro-science toolkit
> >> and currently i use wl_shell exactly because it has the 'driver' method
> >> + refresh rate selection and xdg_shell doesn't, so i'm stuck on wl_shell
> >> until xdg_shell would implement that.
> >
> > Hi Mario,
> >
> 
> Hi Pekka,
> 
> > sorry, I cannot see that happening. We very specifically do not want to
> > give random applications the control of the display beyond their own
> > window. Not in the protocol that will be the basis for all desktops. The
> > design principle for desktops is that the user (person) is in control
> > through the compositor, and random apps can't mess that up even if they
> > tried.
> >
> 
> Not even if the user needs and wants that, in a controlled fashion? I 

If someone comes up with a good proposal for a "controlled fashion",
then I suppose it would be ok. But then you have to think about which
compositor would want to support it.

There has been lots of talk and I think also experiments, but we still
do not have e.g. a mechanism to give privileges to random apps on the
user's consent.

It will be a long road. It also might be out-of-scope for Weston, too,
if it is specifically about the desktop and requires "extensive"
support code in form of GUIs etc.

> understand if you don't want to add features that would undermine 
> security or could allow to mess things up in ways from which the user 
> can't easily recover. Or if one doesn't want to add lots of bloat to 
> common protocol if there isn't any known use case. So maybe providing 
> the full RandR feature set might be too much.
> 
> But the ability for an app to read out or set the gamma table, or 
> control dithering is hardly a security problem, and one could handle it 
> in the same safe way as modesetting in wl_shell's fullscreen DRIVER method.

Sure. A big problem is, where to draw the line on what's ok and what's
not.

Window-specific gamma tables make think of X11's per-window color
maps... though I think we could make it work better.

But, things like gamma table are more a thing related to color
management, so that particular thing would probably be better handled
via color management protocols.

> If an app has an exclusive fullscreen surface on an output, and is 
> thereby the only client displaying anything on that output, allow not 
> only selection of video mode, but also selection of gamma tables. If the 
> app loses fullscreen status, e.g., due to user action, restore to the 
> desktop session default settings. Or define the protocol as hint and 
> leave it to compositor implementations if and how much they honor it? 
> E.g., there could be some security policy to allow or disallow this, or 
> some checkbox in a desktops display control gui a la "Allow regular 
> applications to control display color settings? [x]", or a similar opt-in.

Sure. The implementation and testing effort vs. benefits are a
question, though.

> > I think we have discussed this before, and there are some things we can
> > provide like timings, but absolute control is something we probably
> > won't give for desktop apps. There are conflicting goals here, your use
> > case is not really a desktop use case but something special where the
> > application knows things better than the compositor.
> >
> 
> It's not a typical or trivial desktop app which doesn't care at all how 
> it is displayed, and it does know better than the compositor how it 
> wants to be handled, but it is still mostly a desktop app. The toolkit 
> runs inside a host application which itself has a GUI. In my case that's 
> GNU/Octave with a QT based GUI, or Matlab with a Java/AWT/Swing based 
> GUI. During typical production use the toolkit will take over one or 
> multiple display outputs in fullscreen mode and it needs the ability to 
> at least select video mode (resolution + refresh) and control gamma 
> tables on those fullscreen outputs, ideally also dithering settings. But 
> it's a quite common use case that an operator/experimenter sits in front 
> of another display which displays the Octave/Matlab GUI to interactively 
> monitor and control the experiment session. A small fraction of use 
> cases are run in windowed mode and need a composited desktop to work. 
> And during development and debugging of experiment scripts, one wants to 
> run in windowed mode and have access to the GUI based IDE, editor and 
> debugger. Without a desktop GUI, functionality and usability would be 
> severely restricted and some uses would become impossible.
> 
> I know of multiple similar applications in my field which wouldn't work 
> at all outside a standard desktop GUI, but have the same requirements as 
> mine.
> 
> There are other desktop apps which require a desktop environment and i 
> would imagine also need at least control over video mode and gamma 
> tables, e.g., pro video editing software (think 1 video output for the 
> movie preview, 1 video output for the softwares UI), image 
> editing/desktop publishing/pre-print/color-proofing software, display 
> calibration software, medical visualization applications with a need for 
> controlled display output (radiology etc.). I don't think any of these 
> would be super happy about being restricted to whatever a given version 
> of KDE or GNOME thinks is an appropriate built in tool for this. Even 
> under X i frequently run into bugs and problems which can only be worked 
> around or fixed by use of "xrandr", because none of the official GUI 
> "monitor center" tools of the different X11 desktop environments even 
> exposes the required RandR settings to the user.

The fundamental problem here is that X11 deliberately supports the case
where an application hijacks the whole display server to itself.
Desktop Wayland is designed to never allow that.

You can define extensions to allow that on Wayland, but I'm not sure
any of the big compositors would be willing to implement it - not
without someone else coming up and doing all the work, but it would
still put the review and maintenance burden on the upstream project.
It's the same with Weston, and to limit the maintenance and
review explosion we have defined that vanilla Weston will not provide a
user-friendly desktop.

> > The "driver" method and framerate in the wl_shell fullscreen protocol
> > are defined only as hints. Most compositors would likely ignore them as
> > they can't bother implementing the support or enabling them by default.
> > They were mostly intended as convenience for gamers and game writers:
> > if the user agrees that a game can change the video mode, then it might
> > cause it to happen, based on the compositor user preferences.
> 
> I just hope you will be wrong with nobody bothering to implement those 
> hints, that would be awful for my stuff and for similar pro apps, and 
> also for games. But even then, as long as such properties are at least 
> part of common standard protocol, even if only as hints, one can do 
> something about it, even if that would mean submitting patches myself to 
> each major DE to implement that support. Without a standardized protocol 
> this would become much more painful.

We shall see. It all depends on who is willing to do the work.

> That's why i'd hope at least these options wl_shell would also become 
> part of xdg_shell's fullscreen protocol, even if only as hints.

At the moment, the reasonable thing would be to ensure they can be
added in a later version in backward-compatible way. Xdg-shell has much
bigger problems at the moment, trying to become a standard at all.

> > For a non-desktop case, like the fullscreen shell, it might be
> > possible.
> 
> fullscreen shell -> No way to display desktop apps which weren't 
> designed for it / or XWayland X11 apps i suppose. Also probably no way 
> to configure some compositor outputs for fullscreen shell and some 
> others simultaneously for desktop shell. And i suppose not every major 
> DE like KDE or GNOME will even ship with the option to run fullscreen shell.

All true.

A workaround might be to run one desktop compositor, and one
fullscreen-compositor, and have the app connect to both. It's obviously
far from ideal, and currently would also require two graphics cards.
But, the other graphics card would in the app's total control, could
even be ran directly on KMS.

> > Although, that'd mean replicating much of DRM APIs in the
> > fullscreen shell interfaces, which then raises the question of why
> > bother instead of using KMS directly, if you want your special program
> > to be in full control of the machine.
> 
> "Full control" is relative. X11/GLX + it's various extensions does what 
> i need well enough with mostly good enough performance, so i'd hope 
> Wayland implementations would eventually give me same or better results 
> without too many downsides.

Where the fundamental design principles do not disagree between X11 and
Wayland, yes.

> Using KMS directly would mean implementing and maintaining yet another 
> completely separate display and input backend with > 10000 lines of code 
> in my case. Also KMS currently isn't supported on all platforms and all 
> drivers. I'd be on my own maintaining such a backend... Also i don't 
> think it would be easy or possible at all atm. to have a configuration 
> where the application process which wants a desktop GUI runs inside 
> (X)Wayland on one display and then the process low-level controls a 
> second video output on the same graphics card via drm/kms directly? 
> Afaik, for modesetting a process still needs to be drm-master on a 
> graphics cards /dev/cardX device file, ie. either the compositor or my 
> app, but not both? Running completely without gui wouldn't be 
> impossible, but really painful during typical use - throwing my users 
> back into the golden era of early MS-DOS retro-computing. It would be a 
> last resort for cases which need extreme performance and control, but 
> certainly not what most of my users would want for typical use.

Yeah, KMS resource splitting to separate device nodes is not there
AFAIK, that's why I mentioned two graphics cards above.

> > There actually was an attempt to define and implement a RandR
> > equivalent, but it died on controversy.
> >
> 
> My specific intention here was mostly to raise awareness that some 
> applications really need at least parts of what RandR provided, even if 
> it may need to be implemented in a different or more restricted way to 
> avoid bloat or security holes. I assume that all those X11 extensions 
> didn't grow out of nowhere because somebody was utterly bored and 
> decided to introduce lots of bloat, but there was a need for their 
> functionality and some of that is still needed one way or the other.

Don't forget the difference in the design.

> I assume such needs will show up more frequently when Wayland is in more 
> widespread desktop use and gets exposed to the whole zoo of existing 
> applications.

Sure. But until that happens, we have to solve things like "a way to
define a window in a way that fits both GNOME and KDE", we're not even
there yet.

I guess the take-away from my comments is that I don't see anyone else
pushing for the things you need, and they cannot be implemented the
same way as on X11, hence the negativeness towards a "Wayland RandR". It
is not a single extension you are looking for, but cooperation between
several parts.

Let me try to summarize:
- display timing control: Presentation extension already exists
  (adequate?)
- gamma control: should probably tie in with color management
- dithering: ??
- video mode control: currently no viable proposals, possibly solvable
  via xdg-shell fullscreening protocol with additions (a new window
  state flag maybe).


Thanks,
pq