Dialog hints for shell surfaces

[Martin Novák]

Hello,

I'd like to try to experiment with Wayland/Weston and write a tiling window
manager as a Weston plugin. I was wondering if there's any way for compositor to
tell if the wl_shell_surface or xdg_surface belongs to a `floating window', by
which I mean for example:
	- error information dialog,
	- file chooser dialog,
	- password prompt from email client,
	- Gimp docks (toolbox, layers, ...).

In X, such windows have certain bits of information associated with them, that
make it easy for a tiling WM (such as i3, xmonad, awesome, ...) to treat them as
floating windows implicitly. I assume this is the relevant piece of code from i3
that shows the relevant X bits:

    bool want_floating = false;
    if (xcb_reply_contains_atom(type_reply, A__NET_WM_WINDOW_TYPE_DIALOG) ||
        xcb_reply_contains_atom(type_reply, A__NET_WM_WINDOW_TYPE_UTILITY) ||
        xcb_reply_contains_atom(type_reply, A__NET_WM_WINDOW_TYPE_TOOLBAR) ||
        xcb_reply_contains_atom(type_reply, A__NET_WM_WINDOW_TYPE_SPLASH) ||
        xcb_reply_contains_atom(state_reply, A__NET_WM_STATE_MODAL) ||
        (wm_size_hints.flags & XCB_ICCCM_SIZE_HINT_P_MAX_SIZE &&
         wm_size_hints.flags & XCB_ICCCM_SIZE_HINT_P_MIN_SIZE &&
         wm_size_hints.min_height == wm_size_hints.max_height &&
         wm_size_hints.min_width == wm_size_hints.max_width)) {
        LOG("This window is a dialog window, setting floating\n");
        want_floating = true;
    }

I haven't yet checked the Qt and gtk3 Wayland backends deeply, but since this
kind of information isn't relayed in current Wayland protocols, each toolking
will either define their own or worse, no protocol for it. For instance it seems
that gtk3 maps utility windows (like Gimp docks) either as subsurfaces or as
popups[1], defines own protocol for modal windows[2] and ignores
GDK_WINDOW_TYPE_HINT_DIALOG in Wayland backend.

So it seems there's no reliable information for possible tiling compositor to
differentiate such floating windows. Would it be possible and desirable to
extend the wl_shell_surface or the xdg_surface with requests for setting hints
similar to those in X, so there that there would be a standard interface to
which all toolkits could adhere? I'm of course willing to continue my research
and send patch in future provided I won't be turned down.

Thank you.

[1] https://github.com/GNOME/gtk/blob/197e42efd8a0c6adea9fd49f5469687349e887d4/gdk/wayland/gdkwindow-wayland.c#L1231
[2] https://bugzilla.gnome.org/show_bug.cgi?id=745720