Lots of thoughts on virtual desktop, multi-head, fullscreen, games, video
msbREMOVE-THIS at winterdrache.de
Sat Dec 4 10:39:19 PST 2010
I'm very excited about Wayland and I sincerely hope it'll finally
deliver the features necessary for a game and media center as well as
a good presentation PC.
One of things I've always loved about X (compared to Windows) was
virtual desktops with automatic panning. I can't imagine living without
this feature. Even Full HD doesn't have enough pixels for me
(especially not vertically). I hope that Wayland will support panning
Compared to X there are a few improvements necessary in this area.
Let's take my current setup as an example:
- 2 monitors, running in clone mode: one 1920x1080,
the second (actually a TV) with 1360x768
- a virtual desktop with size 1920 x 1536
This means that both displays will need to pan at some point. The
smaller needs to pan horizontally and vertically. The larger needs to
pan only vertically.
So far this works nicely with X (apart from some mouse cursor bugs).
What doesn't work with X though is full-screen. And this is an area
that Wayland should look at very carefully. Doing full-screen properly
is HARD, especially in multi-head setups.
In my setup there are AFAICS the following different types of
full-screen mode that all make sense. All of them should be supported
by Wayland and the client-side code required to support them all should
be minimal to make sure that they'll always be supported (preferably
they should be supported even in clients without explicit support,
IOW the user should be able to control this without the client's
cooperation if the client doesn't implement support for
the different full-screen modes, e.g. via standard Window decorations):
1. Display the full-screen window on display 1 at that display's
native resolution. DISABLE panning and CONSTRAIN the mouse cursor to
1.1 Keep display 2 panning over a virtual screen with the size of
display 1's native resolution (i.e. 1920x1080).
1.2 Keep display 2 locked to show only part of the window. Sensible
options include showing the center of the window in the center of the
display or showing the upper-left corner in the upper-left corner.
2. Display the full-screen window on display 2 at that display's native
resolution. DISABLE panning and CONSTRAIN the mouse cursor to
2.1 Display the same full-screen window on display 1 (which has a
larger resolution!) centered with black bars on the 4 sides. The mouse
cursor is constrained to the active area.
2.2. Scale the full-screen window up on display 2. Scaler options
should include all of those available in dosbox/mame, including the
high-quality 2D graphics scalers. Either with correct aspect ratio or
stretched to fill the screen.
3. Display the window at the size of the full virtual desktop with
panning on both screens. This is what X does today but it's the least
useful option. It may be useful in some cases, though, such as when
working on a large Freemind mindmap (projected with a beamer and
simultaneously displayed on the notebook's internal screen).
And these are just the options I see as necessary for my setup. That
doesn't include other multi-head setups (such as extended desktop
instead of clone, or even 3-monitor setups). A lot of thought should
go into designing full-screen, multi-head and multi-input for
Wayland, because it's so important for game and multimedia applications.
Just a few random thoughts on this issue:
* In an extended desktop setup (rather than clone) it would be nice to
play a game full-screen on one display (with mouse properly
constrained to the window) while watching a movie full-screen on
another display (that doesn't get the mouse or keyboard events meant
for the game). This should work even if both displays have a
different resolution (like in my setup). It should work even when a
panning virtual desktop is being used (and the panning needs to be
* Task-switching between 2 full-screen apps should be possible. In a
multi-head setup like the previous example it would switch control to
the movie player but keep the displays set up as-is. IOW the displays
wouldn't change but the movie player would get the mouse cursor (of
course constrained to its window) and mouse/keyboard events. Note
that this requires that there be TWO mouse cursor locations for ONE
mouse. Each window has its own cursor (constrained to its bounds) but
the user switches his one mouse between controlling either the one or
* The user must be able to control on which display the application
should go to fullscreen.
* It should be possible to move a full-screen application from one
display to another display with a different resolution. In the
example above I might want to close the movie player and then move
the game to the other display where the movie used to play. The
application needs to be told that its screen area has changed.
I want to stress this point. It is NOT possible for an application to
query the screen size before going fullscreen and expect it to
remain unchanged. It needs to be part of the protocol to account for
changes in size even for full-screen windows.
In fact I think the distinction between fullscreen and non-fullscreen
is one that should be transparent to the application as far as
possible. Of course the application needs to be able to have some
control over this (such as requesting a window to go fullscreen) but
the rest of the application's behaviour, such as which events it gets
etc. should be no different for a windowed or full-screened
* It's implicit in the other examples here but I'd like to
emphasize it: "fullscreen" is a property of a _window_,
not of the _application_. In a multihead setup there can be multiple
fullscreen and non-fullscreen windows at the same time. I really
would like to have the different fullscreen modes as described
further above as options in the window manager's menu where today I
have (in Fluxbox) "Shade", "Maximize", "always-on-top"
etc. That would relieve the clients of the burden of implementing
extra fullscreen support and would make sure that the user always has
all the different options available and is not constrained to a single
"fullscreen" which may or may not be what the user wants (as is the
case today). It has always struck me as odd that Maximize and
Minimize are options provided by the window manager but fullscreen is
an option that has to be provided by the application. I consider
"fullscreen" to be very similar to "maximize" and the code and UI
should reflect that similarity.
* Presentation applications such as Openoffice.org need to be able to
put the presentation full-screen on 1 display
and show notes or other stuff on the other display
(application-controlled, OOo can already do this under X).
* Starting from an extended desktop setup where the desktop is spread
over 2 displays I want to be able to launch a full-screen app (movie
player) on 1 display and keep the rest of the desktop functional on
the second display. Either the desktop must shrink in size (status
bars, docks etc. must shrink accordingly) or it must switch to
panning mode. I want to choose which of these to do. Obviously the
full-screen window should no longer be displayed on the desktop.
Task-switching must take care to constrain the mouse to the
appropriate area when the full-screen app has input focus and to stop
sending events when I switch away from that app to the desktop.
As the example further above, this will require keeping track of TWO
mouse pointer locations even when there is only ONE mouse.
* It should be possible to assign different input devices to different
displays. E.g. when I have a Wiimote driver and a normal
mouse, I want to be able to say that the Wiimote mouse pointer should
be constrained to 1 display while the mouse driver should be
constrained to the other (or should be able to cover both displays).
When a window is full-screen on the Wiimote-controlled display it
would get its events, even when (see previous examples) input focus
for the mouse is on another window.
E.g. I want to have the OpenOffice.org Impress window with the menu
bar (File, Edit, View,...) on one display. It should be
non-full-screen with full-desktop functionality, i.e. I want to be
able to run a text editor or calculator, too, and arrange their
windows as usual. Mouse and keyboard
events should go here. On the other display I want the full-screen
presentation window with a mouse cursor controlled by the Wiimote (as
laser pointer replacement with button presses to move between slides).
* It should be possible to have a different (animated) mouse cursor
for each mouse. E.g. I want a laser-pointer style cursor for my
Wiimote and an arrow for the normal mouse. As the examples above
have shown, a fixed mapping from input device to cursor makes no
sense in a multi-head setup. Even with only ONE mouse I may need to
have TWO mouse cursors (1 constrained to display 1 and the other
constrained to display 2) and switch dynamically which of the
cursors the mouse moves.
A general N-M mapping that can change dynamically is required.
* Clone mode and extended desktop need to be changeable on the fly, as
well as the size (and SHAPE!) of the virtual desktop. In setups with
more than 2 displays, mixing options needs to be possible, such as
having an extended desktop over display 1 and 2 and a panning desktop
on display 3 that clones the whole area.
* A lot of monitors today can be physically rotated. Now imagine what
fun combinations you can do when you have TWO such monitors
connected at the same time. I want to do all of this fun stuff without
restarting Wayland or touching config files.
If Wayland could deliver all of this in a
comfortable to use way it would be the ultimate platform for a media
The important thing here is that all of this needs to be possible
WITHOUT having to quit Wayland, edit config files and restart Wayland.
A lot of this is already possible today with X but it usually requires
a complete restart which closes all applications and requires difficult
editing of xorg.conf and sometimes use of difficult to set up crutches
such as Xnest or Xephyr.
The idea is to build a media-center/game/presentation PC that offers
all of this and is no more difficult to use than a PS3. Switching
from a presentation setup where dad has an image viewer's file manager
on the PC monitor and the full-screen vacation photos displayed on the
home theater's beamer (with mom controlling a red dot with Wiimote) to
a setup where Joey plays Quake 3 full-screen on the PC monitor (with
keyboard/mouse controls) while Angela watches a movie full-screen on the
beamer (controlling the video player with the infrared remote)
needs to be simple and possible without restarting Wayland.
That's all for now. Tell me when you're getting bored and need more
Wenn du einen Burger verspeist, verändert sich nicht der Burger.
Der Burger verändert dich.
More information about the wayland-devel