[PATCH weston 00/17] Follow up from the last xdg-shell series

[Mario Kleiner]

On 04/30/2015 03:48 PM, Pekka Paalanen wrote:
> On Fri, 10 Apr 2015 08:01:39 +0200
> Mario Kleiner <mario.kleiner.de at gmail.com> wrote:
>
>> On 04/07/2015 11:01 AM, wayland-devel-request at lists.freedesktop.org wrote:
>>> Date: Tue,  7 Apr 2015 17:01:15 +0800
>>> From: Jonas Ådahl <jadahl at gmail.com>
>>> To: wayland-devel at lists.freedesktop.org
>>> Cc: Jonas Ådahl <jadahl at gmail.com>
>>> Subject: [PATCH weston 00/17] Follow up from the last xdg-shell series
>>> Message-ID: <1428397292-8455-1-git-send-email-jadahl at gmail.com>
>>> Content-Type: text/plain; charset=UTF-8
>>>
>>> Hi,
>>>
>>>
>>> This series is a follow up mostly addressing the issues raised by Pekka
>>> during the last series as well as some other clarifications.
>>>
>>> The patches them self is a better in depth description of the changes, but
>>> there are a couple of things that I'd like to bring up more explicitly.
>>>
>>> 1.
>>>
>>> This series again bumps the unstable version and breaks backward
>>> compatibility (it moves error enums and introduces a new one).
>>>
>>> If there are any more breaking changes, I think we should
>>> push those together and avoiding unstable version 6. This brings me to
>>> the next point:
>>>
>>> 2.
>>>
>>> Fullscreen as specified in xdg-shell is more limiting than the one in
>>> wl_shell_surface, and it might be a good idea to consider make
>>> fullscreening a bit more powerful in xdg_surface.
>>>
>>> The current difference between wl_shell_surface and xdg_surface in regard
>>> to fullscreening is that while xdg_surface allows for specifying what
>>> output the surface should be fullscreened on, while wl_shell_surface also
>>> allows specifying a method (scale, driver, fill) and for the driver method
>>> a refresh rate.
>>>
>>> As of this series, the method of xdg_surface.set_fullscreen is identical to
>>> t. he wl_shell_surface 'fill' fullscreen method.
>>>
>>> The question is whether a 'scale' and/or a 'driver' method should be made
>>> available.
>>>
>>> Such a 'scale' method would more or less overlap completely with the
>>> wl_scaler protocol, so might be completely unnecessary. Does it provide
>>> any benefit over using wl_scaler?
>>>
>>> The 'driver' method could be good thing to have, assuming it has any
>>> benefit over scaling (i.e. specify refresh rate, (memory bandwidth?)),
>>> as I suspect it wouldn't be any compositors policy to change monitor
>>> resolution for any fullscreen surface where its applicable.
>>>
>>> So the question is, how much, if any, of the fullscreen controls should
>>> be made available via xdg_surface's fullscreen request?
>>>
>>
>> Wrt. the "driver" method and being able to request a refresh rate, yes
>> please! I assume it would also be good for games and other performance
>> hungry apps to avoid wasting gpu cycles / memory bandwidth on scaling
>> etc., but my specific use case is visual presentation software for
>> neuro-science/medical applications. If one wants to find out how eyes
>> and brains work - how we see and perceive and recognize the visual world
>> and also what goes wrong if we don't see that well, also how different
>> senses work together - one needs fine control over video refresh rates,
>> presentation timing and video output resolutions, color reproduction,
>> exact location of pixels on the display etc., often on multi-display
>> setups. Researchers often use precisely calibrated displays, old CRT
>> monitors due to their variable frame rate, or specialized display
>> equipment. For that to work we need control over the exact video mode
>> and refresh rates, if and how a display uses dithering etc. and just
>> rescaling images in the compositor can do rather horrible things to such
>> controlled stimuli. Some of the research equipment parses information
>> that is color-coded in image pixels to trigger hardware operations which
>> are perfectly synchronized to the video stream, so a simple minimal
>> change of pixel location, size or color due to running an unexpected
>> video mode + rescaling and filtering could completely knock out such
>> equipment. Application control over modesetting, at least on displays
>> where the application is fullscreen, is essential for these desktop use
>> cases.
>>
>> To port such applications from X11/GLX to Wayland, ideally i'd like to
>> have the equivalent of as much as possible of X11's full RandR extension
>> - global modesetting, dithering and scaling control, rotation, dynamic
>> changes to color gamma tables etc. But unless/until such things become
>> available, the ability to set a native video mode is a start. I just
>> have started implementing Wayland support for my neuro-science toolkit
>> and currently i use wl_shell exactly because it has the 'driver' method
>> + refresh rate selection and xdg_shell doesn't, so i'm stuck on wl_shell
>> until xdg_shell would implement that.
>
> Hi Mario,
>

Hi Pekka,

> sorry, I cannot see that happening. We very specifically do not want to
> give random applications the control of the display beyond their own
> window. Not in the protocol that will be the basis for all desktops. The
> design principle for desktops is that the user (person) is in control
> through the compositor, and random apps can't mess that up even if they
> tried.
>

Not even if the user needs and wants that, in a controlled fashion? I 
understand if you don't want to add features that would undermine 
security or could allow to mess things up in ways from which the user 
can't easily recover. Or if one doesn't want to add lots of bloat to 
common protocol if there isn't any known use case. So maybe providing 
the full RandR feature set might be too much.

But the ability for an app to read out or set the gamma table, or 
control dithering is hardly a security problem, and one could handle it 
in the same safe way as modesetting in wl_shell's fullscreen DRIVER method.

If an app has an exclusive fullscreen surface on an output, and is 
thereby the only client displaying anything on that output, allow not 
only selection of video mode, but also selection of gamma tables. If the 
app loses fullscreen status, e.g., due to user action, restore to the 
desktop session default settings. Or define the protocol as hint and 
leave it to compositor implementations if and how much they honor it? 
E.g., there could be some security policy to allow or disallow this, or 
some checkbox in a desktops display control gui a la "Allow regular 
applications to control display color settings? [x]", or a similar opt-in.

> I think we have discussed this before, and there are some things we can
> provide like timings, but absolute control is something we probably
> won't give for desktop apps. There are conflicting goals here, your use
> case is not really a desktop use case but something special where the
> application knows things better than the compositor.
>

It's not a typical or trivial desktop app which doesn't care at all how 
it is displayed, and it does know better than the compositor how it 
wants to be handled, but it is still mostly a desktop app. The toolkit 
runs inside a host application which itself has a GUI. In my case that's 
GNU/Octave with a QT based GUI, or Matlab with a Java/AWT/Swing based 
GUI. During typical production use the toolkit will take over one or 
multiple display outputs in fullscreen mode and it needs the ability to 
at least select video mode (resolution + refresh) and control gamma 
tables on those fullscreen outputs, ideally also dithering settings. But 
it's a quite common use case that an operator/experimenter sits in front 
of another display which displays the Octave/Matlab GUI to interactively 
monitor and control the experiment session. A small fraction of use 
cases are run in windowed mode and need a composited desktop to work. 
And during development and debugging of experiment scripts, one wants to 
run in windowed mode and have access to the GUI based IDE, editor and 
debugger. Without a desktop GUI, functionality and usability would be 
severely restricted and some uses would become impossible.

I know of multiple similar applications in my field which wouldn't work 
at all outside a standard desktop GUI, but have the same requirements as 
mine.

There are other desktop apps which require a desktop environment and i 
would imagine also need at least control over video mode and gamma 
tables, e.g., pro video editing software (think 1 video output for the 
movie preview, 1 video output for the softwares UI), image 
editing/desktop publishing/pre-print/color-proofing software, display 
calibration software, medical visualization applications with a need for 
controlled display output (radiology etc.). I don't think any of these 
would be super happy about being restricted to whatever a given version 
of KDE or GNOME thinks is an appropriate built in tool for this. Even 
under X i frequently run into bugs and problems which can only be worked 
around or fixed by use of "xrandr", because none of the official GUI 
"monitor center" tools of the different X11 desktop environments even 
exposes the required RandR settings to the user.

> The "driver" method and framerate in the wl_shell fullscreen protocol
> are defined only as hints. Most compositors would likely ignore them as
> they can't bother implementing the support or enabling them by default.
> They were mostly intended as convenience for gamers and game writers:
> if the user agrees that a game can change the video mode, then it might
> cause it to happen, based on the compositor user preferences.

I just hope you will be wrong with nobody bothering to implement those 
hints, that would be awful for my stuff and for similar pro apps, and 
also for games. But even then, as long as such properties are at least 
part of common standard protocol, even if only as hints, one can do 
something about it, even if that would mean submitting patches myself to 
each major DE to implement that support. Without a standardized protocol 
this would become much more painful.

That's why i'd hope at least these options wl_shell would also become 
part of xdg_shell's fullscreen protocol, even if only as hints.

>
> For a non-desktop case, like the fullscreen shell, it might be
> possible.

fullscreen shell -> No way to display desktop apps which weren't 
designed for it / or XWayland X11 apps i suppose. Also probably no way 
to configure some compositor outputs for fullscreen shell and some 
others simultaneously for desktop shell. And i suppose not every major 
DE like KDE or GNOME will even ship with the option to run fullscreen shell.

Although, that'd mean replicating much of DRM APIs in the
> fullscreen shell interfaces, which then raises the question of why
> bother instead of using KMS directly, if you want your special program
> to be in full control of the machine.

"Full control" is relative. X11/GLX + it's various extensions does what 
i need well enough with mostly good enough performance, so i'd hope 
Wayland implementations would eventually give me same or better results 
without too many downsides.

Using KMS directly would mean implementing and maintaining yet another 
completely separate display and input backend with > 10000 lines of code 
in my case. Also KMS currently isn't supported on all platforms and all 
drivers. I'd be on my own maintaining such a backend... Also i don't 
think it would be easy or possible at all atm. to have a configuration 
where the application process which wants a desktop GUI runs inside 
(X)Wayland on one display and then the process low-level controls a 
second video output on the same graphics card via drm/kms directly? 
Afaik, for modesetting a process still needs to be drm-master on a 
graphics cards /dev/cardX device file, ie. either the compositor or my 
app, but not both? Running completely without gui wouldn't be 
impossible, but really painful during typical use - throwing my users 
back into the golden era of early MS-DOS retro-computing. It would be a 
last resort for cases which need extreme performance and control, but 
certainly not what most of my users would want for typical use.

>
> There actually was an attempt to define and implement a RandR
> equivalent, but it died on controversy.
>

My specific intention here was mostly to raise awareness that some 
applications really need at least parts of what RandR provided, even if 
it may need to be implemented in a different or more restricted way to 
avoid bloat or security holes. I assume that all those X11 extensions 
didn't grow out of nowhere because somebody was utterly bored and 
decided to introduce lots of bloat, but there was a need for their 
functionality and some of that is still needed one way or the other.

I assume such needs will show up more frequently when Wayland is in more 
widespread desktop use and gets exposed to the whole zoo of existing 
applications.

>
> Thanks,
> pq
>