[systemd-devel] /etc/machine-id and a read-only rootfs

[Lennart Poettering]

On Mo, 22.02.21 17:44, Max Marrone (max at opentrons.com) wrote:

> I'm trying to understand some implications of the way systemd loads
> /etc/machine-id.
>
> My understanding is, first: systemd reads /etc/machine-id during *early
> boot.* At that point, I'm only guaranteed that the *root* filesystem will
> be there for me.

Typically that's true. But initrds can of course mount more before
transitioning.

Key really is that /etc/machine-id must be on some storage that is
available dring early boot, right after the initrd→host transition. If
it is on the root fs or actually on some other mounts that are already
assembled in the initrd doesn't matter and up to you.

> Second: there's no way to delay that machine-id initialization until other
> filesystems are ready. It happens early, no matter what.

Yeah. It's something we want userspace to be able to rely on.

> Therefore: if I want a persistent machine-id across reboots, and I want
> systemd to autogenerate it on first boot instead of me predefining it as
> part of installation, then my machine needs the ability to write to its own
> root filesystem.

That's typically how you'd do it, yeah.

> For example, you couldn't have: (1) a permanently *read-only-*mounted root
> filesystem, to which /etc belongs; (2) a separate partition containing a
> writeable machine-id file, which gets bind-mounted atop /etc/machine-id.
> Because the writable one would be available too late in the boot
> process.

The model this is developed for is that you have an immutable /usr,
but /etc is (at least temporarily) writable. i.e. design your system
so that /etc is basically empty at boot. Thankfully today's userspace
largely is fine with empty /etc/, and will use defaults then, or read
fallback configuration from /usr/ somewhere. For the cases where
that's not available you can use tmpfiles.d/ C/L to copy/symlink
minimal configuration files from /usr into /etc, so that everything is
in place.

i.e. have a writable root dir, that comes up initially empty. Then let
PID 1 and tmpfiles do minimal population of it on first boot (or use
tmpfs as root, in which case every boot = first boot).

That way there's very clear distinction of what comes from the vendor
(everything in /usr/) and what is local to the machine (everything
else), and /etc/machine-id is just one part of the latter.

(To extend on this, there are three models this all is designed for:
 1. rootfs+/usr/+/var/ is writable+persistent during runtime, i.e. the
    tradtional Linux setup
 2. rootfs+/usr/ is read-only all the time, /var is writable (and
    usually persistent), and either machine-id pre-initialized
    individually and set up, or transiently randomized on every boot
 3. /usr is read-only all the time, rootfs+/var/ writable during
    runtime (and usually persistent))

> Is all of this correct?
>
> For context, I'm working on an embedded-ish device. We prefer to keep our
> rootfs read-only. All of our writeable stuff, including some system-level
> config files, is in a separate filesystem, accessible from the rootfs by
> symlinks or bind mounts. To update the device's software, we replace the
> whole root partition. This is nice because we never have to worry about an
> update accidentally clobbering some piece of configuration that's meant to
> be persistent. Our problem is that, since our rootfs images have an empty
> /etc/machine-id, a new machine-id gets generated in memory each time the
> device reboots.

I sympathize, but with usr-merge and all the idea really was to do
read-only /usr, and replace that on updates, and have /etc and /var
and root and so on as mutable.

Your model with symlinks and bind mounts of select files in /etc
appears to be a work-around to me, because traditionally software
insisted on putting a bunch of basically static config files in
/etc

> I'm aware of systemd-machine-id-commit.service
> <https://www.freedesktop.org/software/systemd/man/systemd-machine-id-commit.service.html>,
> but that seems to only work for *remounting the rootfs* as read-write, not
> using a different filesystem entirely. It would let me write-back the
> machine-id when that filesystem becomes writable, but it wouldn't let me
> postpone reading the machine-id until that filesystem is mounted. So
> systemd would still generate a new machine-id on each reboot, I think.

You can also pre-initialize /etc/machine-id from the initrd, right
before transitioning to the real OS, by overmounting it. If it's
already initialized PID 1 will happy use it after all.

Lennart

--
Lennart Poettering, Berlin