[systemd-devel] Using systemd.offline-updates from an ostree based system

[Dan Nicholson]

On Fri, Apr 26, 2019 at 3:54 AM Lennart Poettering
<lennart at poettering.net> wrote:
>
> On Do, 25.04.19 17:10, Richard Hughes (hughsient at gmail.com) wrote:
>
> >  Hi all,
> >
> > I use the offline updates feature
> > https://www.freedesktop.org/software/systemd/man/systemd.offline-updates.html
> > in fwupd to install some kinds of firmware. I've just found out this
> > doesn't work on Fedora SilverBlue as / is immutable and of course
> > creating the /system-update symlink fails.
>
> Hmm, the assumption was always that / was mutable if offline updates
> are used to update /...
>
> I have the suspicion the ostree folks might have some specific ideas
> on how to do this in their environment?

In a bootable ostree you're never updating /. You're updating a new
deployment in /ostree/deploy/<os>/deploy/<deployment>. In a standard
update path you'd make modifications in there, but this is different
since you're not updating the OS.

> > I'm using the offline update mode rather than updating "live" as I'm
> > updating the keyboard controller itself, which means the keyboard (and
> > power button!) is disabled and various things like ACPI battery
> > reporting also get upset for the 30 seconds of flashing. I'm stuck for
> > ideas. I suppose I could write a new boot entry (BLS?) that has a new
> > kernel command line argument set to go into system-update.target (and
> > patch src/system-update-generator/system-update-generator.c) although
> > I don't think BLS has the concept of a "oneshot" entry that would
> > delete after running it once.
>
> What precisely is writable on those systems? Is /etc? You need
> something that is accessible during early boot (i.e. right after the
> transition from the initrd to the host OS). This excludes /var in the
> general case, but includes /etc, / or /usr...

/etc and /var are the typical places to put writable data (/usr is
also mounted RO). Early in boot a writable /etc is available.
Basically, there are 2 parts to the ostree boot process.

In the initrd, ostree-prepare-root runs late to take the deployment in
/sysroot/ostree/<os>/deploy/<deployment> and remounts things so that
it sits at /sysroot instead. It does most of the mounting to prepare
the system including making /usr a read only mount. It will mount
/sysroot/ostree/<os>/var (a /var shared between deployments of the
same OS) at /var in some scenarios but not all. It also handles bind
mounting /boot to the correct location if it's on the same filesystem
as the deployments. See the following:

https://github.com/ostreedev/ostree/blob/master/src/switchroot/ostree-prepare-root.c
https://github.com/ostreedev/ostree/blob/master/src/boot/ostree-prepare-root.service

Note that ostree makes a separate reference to "/sysroot". At runtime,
the actual root of the filesystem is mounted at /sysroot and there are
a couple symlinks into it to make things work. For instance, /ostree
is a symlink to /sysroot/ostree. Creating new deployments on the live
system wouldn't work if /ostree was actually a directory inside the
currently running deployment.

After pivoting to the new root, there's a generator that runs that
ensures /var is mounted as described above. It's in 2 pieces at:

https://github.com/ostreedev/ostree/blob/master/src/switchroot/ostree-system-generator.c
https://github.com/ostreedev/ostree/blob/master/src/libostree/ostree-impl-system-generator.c

> Note that the boot loader spec is not universally adopted. Fedora uses
> something they call the boot loader spec, but it#s very different from
> what bootctl, logind, systemd's shutdown code and sd-boot understand
> and not compatible. They really should stop using the name in its
> current state, it's just confusing.
>
> The boot loader interface systemd undestands actually has support for
> oneshot boots, see:
>
> https://systemd.io/BOOT_LOADER_INTERFACE.html, see the
> LoaderEntryOneShot EFI variable.
>
> Given that GRUB doesn't support that and maybe you don't want to bind
> yourself to EFI variables it's not the best approach anyway?
>
> I mean, you could store the flag at the following places at least:
>
>    1. EFI variables
>    2. ESP
>    3. /boot if different from ESP
>    4. /
>    5. /etc

I think /etc is the only guaranteed to be writable location that's
generic to all ostree systems. If possible, I'd get systemd to honor
/etc/system-update.

--
Dan