[systemd-devel] Setting up a VPN daemon as a Portable Service

[Luca Boccassi]

On Mon, 10 Oct 2022 at 04:00, Duncan Gibson <legowerewolf at gmail.com> wrote:
>
> Final update, hopefully. Here's a gist with a script, service unit, and readme.

Again, that is not safe and it will fail at some point as it is open
to race conditions. You have to use ExtensionImages= instead.

> On Sun, Oct 9, 2022 at 11:10 AM Duncan Gibson <legowerewolf at gmail.com> wrote:
>>
>> After doing some more looking, it seems like the /etc folder is overlaid with /var/lib/overlays/etc/upper, meaning that changes to /etc/ get saved in the overlay, which should survive updates. I've added the service definition there and the binaries to my home directory (and updated the service definition to point there), and it seems to just work. I might look into putting the binaries in a system extension, and update the service definition to require the extension be running, but it seems to work as-is. Time to wait for the next system update and see if it breaks.
>>
>> On Sat, Oct 8, 2022 at 2:02 PM Luca Boccassi <bluca at debian.org> wrote:
>>>
>>> On Sat, 8 Oct 2022 at 18:51, Duncan Gibson <legowerewolf at gmail.com> wrote:
>>> >
>>> > Hm. Actually, no, I don't think it will. Services installed the normal way won't survive the A/B update system.
>>>
>>> I don't know what that is and how it works.
>>>
>>> > On Sat, Oct 8, 2022 at 12:23 PM Duncan Gibson <legowerewolf at gmail.com> wrote:
>>> >>
>>> >> Oh, now that's a new way of doing it. I'll definitely give that a shot. That sounds like it has the best chance of working.
>>> >>
>>> >> On Sat, Oct 8, 2022 at 12:20 PM Luca Boccassi <bluca at debian.org> wrote:
>>> >>>
>>> >>> On Sat, 2022-10-08 at 11:13 -0400, Duncan Gibson wrote:
>>> >>> > The problem wasn't mounting the system extension automatically. That
>>> >>> > worked
>>> >>> > just fine. It was that systemd would try to start the service before
>>> >>> > the
>>> >>> > system extension mounted, which would fail, for obvious reasons. This
>>> >>> > weekend I think I'm going to try the BindReadOnlyPaths option and see
>>> >>> > if I
>>> >>> > can get that to work.
>>> >>>
>>> >>> Don't do that. system-wide extensions are not supposed to add units,
>>> >>> and it will not work. Portable services are for distributors - for
>>> >>> locally built extensions, you can simply use a normal service with
>>> >>> ExtensionImages= that points to your extension, and it will be
>>> >>> overlayed with the rootfs.
>>> >>>
>>> >>> > On Fri, Oct 7, 2022 at 3:35 PM David Anderson <dave at natulte.net>
>>> >>> > wrote:
>>> >>> >
>>> >>> > > Yeah, so far we (tailscale) haven't found a good way to run on the
>>> >>> > > Steam
>>> >>> > > Deck at bootup, and also survive the A/B OS updates. Systemd system
>>> >>> > > extensions _can_ be activated during bootup, if you place the
>>> >>> > > extension in
>>> >>> > > one of the well-known locations (/var/lib/extensions would be the
>>> >>> > > one to
>>> >>> > > use on Deck, as iirc it survives A/B upgrades), and the systemd-
>>> >>> > > sysext
>>> >>> > > service is enabled.
>>> >>> > >
>>> >>> > > I would check if systemd-sysext.service is enabled on the deck, and
>>> >>> > > if
>>> >>> > > not, file a request with Valve to enable that service in a future
>>> >>> > > update.
>>> >>> > > You should present it as enabling further customization of their
>>> >>> > > platform.
>>> >>> > >
>>> >>> > > Another possible reason that sysexts aren't working for you, is
>>> >>> > > that the
>>> >>> > > Deck's /etc/os-release doesn't define a SYSEXT_LEVEL, and the
>>> >>> > > VERSION_ID
>>> >>> > > changes with every OS update. Because of this, the system extension
>>> >>> > > will
>>> >>> > > refuse to activate after every update (either SYSEXT_LEVEL or
>>> >>> > > VERSION_ID
>>> >>> > > must match exactly), until you rebuild a new image with the right
>>> >>> > > OS
>>> >>> > > metadata. Asking Valve to set SYSEXT_LEVEL to a stable value would
>>> >>> > > make it
>>> >>> > > even easier to provide Deck OS extensions reliably :)
>>> >>> > >
>>> >>> > > - Dave
>>> >>> > >
>>> >>> > > On Thu, Oct 6, 2022, at 12:08, Arian van Putten wrote:
>>> >>> > >
>>> >>> > > Afaik Portable services run in an isolated root and dont have
>>> >>> > > access to
>>> >>> > > the hosts rootfs.  You'd have go include iptables and all its
>>> >>> > > dependencies
>>> >>> > > in the portable services directory. If you don't want to do that
>>> >>> > > you'd have
>>> >>> > > to use BindReadOnlyPaths= to give the service access to the
>>> >>> > > required host
>>> >>> > > paths or you'd have to use a system extension.
>>> >>> > >
>>> >>> > > That's probably why they advice running as a system extension.
>>> >>> > >
>>> >>> > > I think there are mechanisms for setting up system extensions on
>>> >>> > > startup
>>> >>> > > but I'm not familiar enough with the details. Maybe someone else in
>>> >>> > > the
>>> >>> > > list knows.
>>> >>> > >
>>> >>> > >
>>> >>> > >
>>> >>> > >
>>> >>> > > On Thu, 6 Oct 2022, 20:21 Duncan Gibson, <legowerewolf at gmail.com>
>>> >>> > > wrote:
>>> >>> > >
>>> >>> > > Hi, everyone.
>>> >>> > >
>>> >>> > > The high-level overview: I'm trying to install Tailscale as a
>>> >>> > > portable
>>> >>> > > service on my Steam Deck.
>>> >>> > >
>>> >>> > > Tailscale is a point-to-point VPN service, essentially a wrapper
>>> >>> > > around
>>> >>> > > Wireguard that helps with network setup and management. The Steam
>>> >>> > > Deck is
>>> >>> > > Valve's handheld PC running SteamOS 3, which is derived from Arch.
>>> >>> > > It uses
>>> >>> > > an A/B partition system for system files, meaning you can't install
>>> >>> > > a
>>> >>> > > service the normal way.
>>> >>> > >
>>> >>> > > There *is* a guide to do this, posted on their own blog, but it
>>> >>> > > uses
>>> >>> > > system extensions which aren't good for services that you want to
>>> >>> > > run on
>>> >>> > > startup. Indeed, following that guide puts me in a state where I
>>> >>> > > have to
>>> >>> > > manually start the daemon every time I reboot my Deck, even with
>>> >>> > > the
>>> >>> > > service enabled.
>>> >>> > >
>>> >>> > > Let's move on to how I've started to do this.
>>> >>> > >
>>> >>> > > Tailscale is available through most package managers, but they also
>>> >>> > > publish static binaries with systemd unit files.
>>> >>> > >
>>> >>> > > This script grabs that binary, extracts it, and moves it into a
>>> >>> > > portable
>>> >>> > > service directory structure.
>>> >>> > >
>>> >>> > > # download and extract Tailscale
>>> >>> > > tarball="$(curl -s 'https://pkgs.tailscale.com/stable/?mode=json' |
>>> >>> > > jq -r
>>> >>> > > .Tarballs.amd64)"
>>> >>> > > version="$(echo ${tarball} | cut -d_ -f2)"
>>> >>> > > tar_dir="$(echo ${tarball} | cut -d. -f1-3)"
>>> >>> > > curl -s "https://pkgs.tailscale.com/stable/${tarball}" -o
>>> >>> > > tailscale.tgz
>>> >>> > > tar xzf tailscale.tgz
>>> >>> > > test -d $tar_dir
>>> >>> > >
>>> >>> > > # Set up our target directory structure
>>> >>> > > mkdir -p
>>> >>> > > tailscaled/{usr/{bin,sbin,lib/systemd/system},etc,proc,sys,dev,run,
>>> >>> > > /var/tmp}
>>> >>> > >
>>> >>> > > # Copy tailscale-distributed files to the right place
>>> >>> > > cp -rf $tar_dir/tailscaled tailscaled/usr/sbin/tailscaled
>>> >>> > > cp -rf $tar_dir/systemd/tailscaled.service
>>> >>> > > tailscaled/usr/lib/systemd/system/tailscaled.service
>>> >>> > >
>>> >>> > > # Write service os-release file
>>> >>> > > source /etc/os-release
>>> >>> > > cp -rf /etc/os-release tailscaled/etc/os-release
>>> >>> > >
>>> >>> > > Not automated yet is patching the provided unit file - you need to
>>> >>> > > remove
>>> >>> > > the EnvironmentFile line and "--port $PORT $FLAGS" options, and add
>>> >>> > > [Exec]
>>> >>> > > Environment="PATH=/usr/bin"
>>> >>> > >
>>> >>> > > Attach the portable service: sudo portablectl attach ./tailscaled
>>> >>> > > --profile=trusted
>>> >>> > > and try starting it: sudo systemctl start tailscaled
>>> >>> > >
>>> >>> > > It fails, leaving this in the logs:
>>> >>> > >
>>> >>> > > logtail started
>>> >>> > > Program starting: v1.30.2-t24c524c78-gc399ae6fa, Go 1.19.1-
>>> >>> > > tsb13188dd36:
>>> >>> > > []string{"/usr/sbin/tailscaled",
>>> >>> > > "--state=/var/lib/tailscale/tailscaled.state",
>>> >>> > > "--socket=/run/tailscale/tailscaled.sock"}
>>> >>> > > LogID:
>>> >>> > > 0f59ed267a2b19cc28aac9ee7119914000ca478234af8d56893a025ae72cc647
>>> >>> > > logpolicy: using $STATE_DIRECTORY, "/var/lib/tailscale"
>>> >>> > > wgengine.NewUserspaceEngine(tun "tailscale0") ...
>>> >>> > > wgengine.NewUserspaceEngine(tun "tailscale0") error: creating
>>> >>> > > router:
>>> >>> > > could not get iptables version: fork/exec /usr/bin/iptables: no
>>> >>> > > such file
>>> >>> > > or directory flushing log.
>>> >>> > > logger closing down
>>> >>> > > createEngine: creating router: could not get iptables version:
>>> >>> > > fork/exec
>>> >>> > > /usr/bin/iptables: no such file or directory
>>> >>> > >
>>> >>> > > iptables is, in fact, at /usr/bin/iptables, so what am I missing?
>>> >>> > > Before I
>>> >>> > > added the Environment line, I was getting errors that iptables
>>> >>> > > wasn't on
>>> >>> > > the PATH, so I suspect that now tailscaled can *see* iptables, but
>>> >>> > > systemd isn't letting tailscaled run it.
>>> >>> > >
>>> >>> > > Thanks for having a look at this.
>>> >>> > >
>>> >>> > >
>>> >>> > >
>>> >>>
>>> >>> --
>>> >>> Kind regards,
>>> >>> Luca Boccassi