Lots of thoughts on virtual desktop, multi-head, fullscreen, games, video

[Matthias B.]

Hi,

I'm very excited about Wayland and I sincerely hope it'll finally
deliver the features necessary for a game and media center as well as
a good presentation PC.

One of things I've always loved about X (compared to Windows) was
virtual desktops with automatic panning. I can't imagine living without
this feature. Even Full HD doesn't have enough pixels for me
(especially not vertically). I hope that Wayland will support panning
virtual screens.

Compared to X there are a few improvements necessary in this area.
Let's take my current setup as an example:

- 2 monitors, running in clone mode: one 1920x1080, 
the second (actually a TV) with 1360x768

- a virtual desktop with size 1920 x 1536

This means that both displays will need to pan at some point. The
smaller needs to pan horizontally and vertically. The larger needs to
pan only vertically.

So far this works nicely with X (apart from some mouse cursor bugs).
What doesn't work with X though is full-screen. And this is an area
that Wayland should look at very carefully. Doing full-screen properly
is HARD, especially in multi-head setups. 

In my setup there are AFAICS the following different types of
full-screen mode that all make sense. All of them should be supported
by Wayland and the client-side code required to support them all should
be minimal to make sure that they'll always be supported (preferably
they should be supported even in clients without explicit support,
IOW the user should be able to control this without the client's
cooperation if the client doesn't implement support for
the different full-screen modes, e.g. via standard Window decorations):

1. Display the full-screen window on display 1 at that display's
native resolution. DISABLE panning and CONSTRAIN the mouse cursor to
the window.

1.1 Keep display 2 panning over a virtual screen with the size of
display 1's native resolution (i.e. 1920x1080).

1.2 Keep display 2 locked to show only part of the window. Sensible
options include showing the center of the window in the center of the
display or showing the upper-left corner in the upper-left corner.

2. Display the full-screen window on display 2 at that display's native
resolution. DISABLE panning and CONSTRAIN the mouse cursor to
the window.

2.1 Display the same full-screen window on display 1 (which has a
larger resolution!) centered with black bars on the 4 sides. The mouse
cursor is constrained to the active area.

2.2. Scale the full-screen window up on display 2. Scaler options
should include all of those available in dosbox/mame, including the
high-quality 2D graphics scalers. Either with correct aspect ratio or
stretched to fill the screen.

3. Display the window at the size of the full virtual desktop with
panning on both screens. This is what X does today but it's the least
useful option. It may be useful in some cases, though, such as when
working on a large Freemind mindmap (projected with a beamer and
simultaneously displayed on the notebook's internal screen).


And these are just the options I see as necessary for my setup. That
doesn't include other multi-head setups (such as extended desktop
instead of clone, or even 3-monitor setups). A lot of thought should
go into designing full-screen, multi-head and multi-input for
Wayland, because it's so important for game and multimedia applications.

Just a few random thoughts on this issue:

* In an extended desktop setup (rather than clone) it would be nice to
  play a game full-screen on one display (with mouse properly
  constrained to the window) while watching a movie full-screen on
  another display (that doesn't get the mouse or keyboard events meant
  for the game). This should work even if both displays have a
  different resolution (like in my setup). It should work even when a
  panning virtual desktop is being used (and the panning needs to be
  disabled temporarily).

* Task-switching between 2 full-screen apps should be possible. In a
  multi-head setup like the previous example it would switch control to
  the movie player but keep the displays set up as-is. IOW the displays
  wouldn't change but the movie player would get the mouse cursor (of
  course constrained to its window) and mouse/keyboard events. Note
  that this requires that there be TWO mouse cursor locations for ONE
  mouse. Each window has its own cursor (constrained to its bounds) but
  the user switches his one mouse between controlling either the one or
  the other.

* The user must be able to control on which display the application
  should go to fullscreen.

* It should be possible to move a full-screen application from one
  display to another display with a different resolution. In the
  example above I might want to close the movie player and then move
  the game to the other display where the movie used to play. The
  application needs to be told that its screen area has changed.
  I want to stress this point. It is NOT possible for an application to
  query the screen size before going fullscreen and expect it to
  remain unchanged. It needs to be part of the protocol to account for
  changes in size even for full-screen windows.
  In fact I think the distinction between fullscreen and non-fullscreen
  is one that should be transparent to the application as far as
  possible. Of course the application needs to be able to have some
  control over this (such as requesting a window to go fullscreen) but
  the rest of the application's behaviour, such as which events it gets
  etc. should be no different for a windowed or full-screened
  application.

* It's implicit in the other examples here but I'd like to
  emphasize it: "fullscreen" is a property of a _window_,
  not of the _application_. In a multihead setup there can be multiple
  fullscreen and non-fullscreen windows at the same time. I really
  would like to have the different fullscreen modes as described
  further above as options in the window manager's menu where today I
  have (in Fluxbox) "Shade", "Maximize", "always-on-top"
  etc. That would relieve the clients of the burden of implementing
  extra fullscreen support and would make sure that the user always has
  all the different options available and is not constrained to a single
  "fullscreen" which may or may not be what the user wants (as is the
  case today). It has always struck me as odd that Maximize and
  Minimize are options provided by the window manager but fullscreen is
  an option that has to be provided by the application. I consider
  "fullscreen" to be very similar to "maximize" and the code and UI
  should reflect that similarity.

* Presentation applications such as Openoffice.org need to be able to
  put the presentation full-screen on 1 display
  and show notes or other stuff on the other display
  (application-controlled, OOo can already do this under X).

* Starting from an extended desktop setup where the desktop is spread
  over 2 displays I want to be able to launch a full-screen app (movie
  player) on 1 display and keep the rest of the desktop functional on
  the second display. Either the desktop must shrink in size (status
  bars, docks etc. must shrink accordingly) or it must switch to
  panning mode. I want to choose which of these to do. Obviously the
  full-screen window should no longer be displayed on the desktop.
  Task-switching must take care to constrain the mouse to the
  appropriate area when the full-screen app has input focus and to stop
  sending events when I switch away from that app to the desktop.
  As the example further above, this will require keeping track of TWO
  mouse pointer locations even when there is only ONE mouse.

* It should be possible to assign different input devices to different
  displays. E.g. when I have a Wiimote driver and a normal
  mouse, I want to be able to say that the Wiimote mouse pointer should
  be constrained to 1 display while the mouse driver should be
  constrained to the other (or should be able to cover both displays).
  When a window is full-screen on the Wiimote-controlled display it
  would get its events, even when (see previous examples) input focus
  for the mouse is on another window.
  E.g. I want to have the OpenOffice.org Impress window with the menu
  bar (File, Edit, View,...) on one display. It should be
  non-full-screen with full-desktop functionality, i.e. I want to be
  able to run a text editor or calculator, too, and arrange their
  windows as usual. Mouse and keyboard
  events should go here. On the other display I want the full-screen
  presentation window with a mouse cursor controlled by the Wiimote (as
  laser pointer replacement with button presses to move between slides).

* It should be possible to have a different (animated) mouse cursor
  for each mouse. E.g. I want a laser-pointer style cursor for my
  Wiimote and an arrow for the normal mouse. As the examples above 
  have shown, a fixed mapping from input device to cursor makes no
  sense in a multi-head setup. Even with only ONE mouse I may need to
  have TWO mouse cursors (1 constrained to display 1 and the other
  constrained to display 2) and switch dynamically which of the
  cursors the mouse moves.
  A general N-M mapping that can change dynamically is required.

* Clone mode and extended desktop need to be changeable on the fly, as
  well as the size (and SHAPE!) of the virtual desktop. In setups with
  more than 2 displays, mixing options needs to be possible, such as
  having an extended desktop over display 1 and 2 and a panning desktop
  on display 3 that clones the whole area.

* A lot of monitors today can be physically rotated. Now imagine what
  fun combinations you can do when you have TWO such monitors
  connected at the same time. I want to do all of this fun stuff without
  restarting Wayland or touching config files.


If Wayland could deliver all of this in a
comfortable to use way it would be the ultimate platform for a media
center PC.

The important thing here is that all of this needs to be possible
WITHOUT having to quit Wayland, edit config files and restart Wayland.
A lot of this is already possible today with X but it usually requires
a complete restart which closes all applications and requires difficult
editing of xorg.conf and sometimes use of difficult to set up crutches
such as Xnest or Xephyr.

The idea is to build a media-center/game/presentation PC that offers
all of this and is no more difficult to use than a PS3. Switching
from a presentation setup where dad has an image viewer's file manager
on the PC monitor and the full-screen vacation photos displayed on the
home theater's beamer (with mom controlling a red dot with Wiimote) to
a setup where Joey plays Quake 3 full-screen on the PC monitor (with
keyboard/mouse controls) while Angela watches a movie full-screen on the
beamer (controlling the video player with the infrared remote)
needs to be simple and possible without restarting Wayland.

That's all for now. Tell me when you're getting bored and need more
challenges ;-)

MSB

-- 
Wenn du einen Burger verspeist, verändert sich nicht der Burger.
Der Burger verändert dich.