Chrome Remote Desktop and Wayland

Fri Apr 24 14:13:35 UTC 2020

Hi,

On Thu, Apr 23, 2020 at 2:27 AM Jonas Ådahl <jadahl at gmail.com> wrote:
> Can't the remote login session still be "wayland", but without being
> able to be drm master?
So I think you're saying:

"Clients still speak the wayland protocol when talking to the
display server even if they're getting displayed via pipewire,
so they should still have a wayland session type."

It's a valid point that the possible session types are named
after display server protocols, not display server outputs.
i.e., we have "x11" and "wayland" not "x11-over-kms" and
"wayland-over-kms". Although it all gets muddy when you consider
mir supports wayland protocols, and even wayland supports x11
protocols with Xwayland, etc.

> We still need API to manage the pipewire streams
> (add/remove/change virtual outputs,
So in my proposal, I said a virtual output can be automatically
created when the logind session is registered, and all windows
can be moved to that output when the session switches from
"online" to "active" state (and we'd add some way for logind to
be able to be told to move from "online" to "active" in this new
session type when a user connects)

> inject input,
So this point here is a strong reason not to have a session type
named "pipewire".  And it's not something I'd considered before.
I mean input isn't going through pipewire. And, of course, even
getting the pipewire stream needs to go through mutter. I guess
pipewire isn't really first class, or distinguishing here.

So you've convinced me.

> would adding a new session type bring us anything useful?
Well, it would be useful if we had some logind level bookkeeping
that distinguish e.g., vnc sessions from drm ones.

but I guess session type isn't going to work out for that
purpose. Maybe, sd_session_get_service is good enough for that
purpose, then.  logind sessions are generally registered with
pam_systemd anyway, so there's going to be an associated pam
service file.

If we decide it's not good enough, we could always try to add a

sd_session_get_output_target (render_platform/medium/whatever)

> We wouldn't just implicitly spit out a
> pipewire stream for some made up output, I assume we still want it to be
> managed some how by the remote desktop service.

Okay, so let's do another iteration, and talk this through this
updated vision a little deeper but from the gnome-remote-desktop
point of view.  I assume chrome-remote-desktop would be similar
but let's start with g-r-d.  If this is TL;DR territory let me know and
I'll throw it on the GNOME wiki instead.

Here's how I believe gnome-remote-desktop works *today* with a
shared local session instead of a curtained local session:

0. User logs in from the GDM login screen on VT 1. This opens a
pam session, registers a logind session, and runs gnome-session

1. gnome-session starts the session off by doing:

$ systemctl --user start gnome-session-wayland at gnome.target

where "wayland" is the session type (the @gnome part isn't
important for this discussion, but it could also be
"@gnome-classic" for classic mode or "@gnome-login" for the
login screen).

2. This target pulls in gnome-shell-wayland.service which starts
gnome-shell.

3. Gnome-shell initializes its "native" backend where it sets up
kernel modesetting.  It uses the graphics hardware attached to
seat0.  If the oldest graphical session isn't attached to seat0,
gnome-shell will fail to initialize.  This logic will obviously
need to be improved.

4. The gnome-session-wayland target also pulls in
gsd-sharing.service which starts the daemon that starts
gnome-remote-desktop. This is guaranteed to be started after
gnome-shell is started.

5. gnome-remote-desktop tells mutter it's going to be doing
input remotely by calling
org.gnome.Mutter.RemoteDesktop.CreateSession. This makes mutter
set up virtual input devices, and provides access to those input
devices via a unique identifier.

6. gnome-remote-desktop tells mutter it's going to be doing
output remotely by calling
org.gnome.Mutter.ScreenCast.CreateSession.  It links that output
to the previously set up virtual input devices by passing the
unique identifier generated in step 5 to this call.

7. gnome-remote-desktop tells mutter to send the primary monitor
of the already running session to the newly created screencast
session by calling
org.gnome.Mutter.ScreenCast.Session.RecordMonitor.  This call
returns a pipewire stream, with one end hooked up to the primary
monitor of the already running session.

8. gnome-remote-desktop acts as a VNC server and forwards all
input from VNC clients to mutter via the virtual input devices
created in step 5.  It pushes video from the pipewire stream
created in step 7 to VNC clients.

I think I got those steps right, but tell me if you spot anything
I got wrong. :-).

Now how would gnome-remote-desktop go from here to a world with
curtaining?

0.  We're no longer going to have one session that gets shared,
so we need two distinct sessions registered. One session is
still the user logging in from the local GDM screen, but the
other "headless" one will have to be some system service that
can register the new session with logind (via pam_systemd) and
fire it off. I imagine from an GNOME point of view GDM would
serve this role, but chrome-remote-desktop could do it as well.

1. gnome-session on the local session does

$ systemctl --user start gnome-session-wayland at gnome.target

just as before.  gnome-session on the headless session also does

$ systemctl --user start gnome-session-wayland at gnome.target

2. In both cases the target pulls in gnome-shell, but the second
one to run is a noop with regard to gnome-shell because it's already
running from the first call. gnome-shell is running as a sort of
"factory" at this point

3. when gnome-shell is started it initializes its "native"
backend.  It doesn't actually do any modesetting until the local
session is registered.  When the local session is registered, it
notices that the session is associated with seat0 and so can do
a modeset on the drm device associated with that seat.

When the headless session is registered it notices it has no
associated seat, and so no associated drm device, so it
doesn't do much.

Crucially, though, whenever a new logind session is registered
(be it headless or local), the shell should lock the screen.

4. Just as before gsd-sharing is started.  Just like
gnome-shell, it's a single instance factory. The first session
started actually triggers gsd-sharing getting started, but the
other one is a noop. When just the local session is registered,
gsd-sharing doesn't start gnome-remote-desktop, but once the
headless session appears gnome-remote-desktop gets started. This
is because in this line of thought, GNOME isn't configured to
share the local session.

5. gnome-remote-desktop starts up and checks the list of
sessions registered by the user.  If it sees one with the
correct service name (or output target if we get logind changed
to have that), then, like before, needs to create a remote
desktop session to set up virtual input devices.  But unlike
before, it also needs to set up a virtual output device.  You probably
have opinions on what the API should look like, but I'll just throw
out there one possibility: we could
keep org.gnome.Mutter.RemoteDesktop.CreateSession and add a new
org.gnome.Mutter.RemoteDesktop.CreateOutput that gets called
afterward.

6. Again, like before, gnome-remote-desktop calls
org.gnome.Mutter.ScreenCast.CreateSession to tell mutter it's
going to be doing output, and links it to the previously set up
input devices from step 5. But now this step also links in the
new output device

7. Instead of calling
org.gnome.Mutter.ScreenCast.Session.RecordMonitor here, g-r-d would
call a new org.gnome.Mutter.ScreenCast.Session.RecordOutput. This
call would return a pipewire stream with one end hooked up to the
virtual output created in step 5.

8. As I mentioned in step 3, when a new logind session is
registered, the shell should lock.  The newly created virtual
output will also be locked.

9. Like before gnome-remote-desktop acts as a VNC server and
forwards all input from VNC clients to mutter via the virtual
input devices created in step 5.  It pushes video from the
pipewire stream created in step 7 to VNC clients.

10. Windows should automatically be migrated to whatever session
gets unlocked by the user.  When a particular session goes from
"active" to "online" that session should automatically lock.

11. Ideally, we'd have the remote session go from "active" to
"online" anytime the user disconnects their vnc client.  This
will need some changes to logind.  But failing that we could
also just forcefully lock the session from gnome-remote-desktop
anytime a user disconnects.  It just means loginctl won't let us
easily see the connected status of remote sessions.

Sound reasonable?
Ray