[systemd-devel] Stateless system - Centos8 and overlayfs on root

[Lennart Poettering]

On Fr, 27.03.20 10:17, Preston L. Bannister (preston at bannister.us) wrote:

> Looking for a sanity check from the folk how know more of systemd than do
> I. Not looking for someone else to solve my problem, but could use a clue.
>
> Trying to figure out how to get an overlayfs root mounted early in boot.
>
> Building an embedded system that must not store any sensitive data when
> powered down. Found this was referred to on this list more recently as a
> "stateless" system. (And Tobias Hunger seems to have the most to say. :) )
> We are using Centos8, with systemd 239. Starting point is a vanilla Centos8
> server installation to a small (8 or 32GB) flash volume.
>
> Had the (possibly) clever notion of using an overlayfs as the root mount,
> with a tmpfs as the upper, and the usual persistent volume as the
> lower.

Current systemd versions support this natively, just boot with
"systemd.volatile=overlay" on the kernel cmdline. Doubt this is in
centos8 yet, though.

Note that overlayfs is a weird fs, it has strange, non-posixy
semantics (inode nrs a fucked). It generally doesn't work as well as
people want it to work, and while you might get away with using it for
small, well-defined scenarios it's not suitable for complex, general
purpose systems to run as root fs.

"systemd.volatile=overlay" is a nice tool for testing and development,
and maybe some very specific setups, but it's not really something i'd
want to deploy in production in big scale.

> The  initial round of configuration and test would just be against a stock
> Centos8 install. Once fully configured and tested, would add a default boot
> menu item to boot with root mounted as an overlayfs, with the fully
> configured root volume as the read-only lower.
>
> Updates would be accomplished by booting from the original boot menu entry.
> (This is slightly complicated by the fact the target systems' computers do
> not have a console - but figure I can script altering the default
> boot.)

I doubt overlayfs is good enough to make this workable. In particular
not in the old version included in rhel8...

> BASE=/run/overlay_root
> OVERLAY=$BASE/merged
> ROOT_MOUNT=$OVERLAY/mnt/root
> mount -t tmpfs root-base $BASE ; mkdir $BASE/{lower,upper,work,merged}
> mount --bind / $BASE/lower
> mount -t overlayfs root-overlay $OVERLAY
> -olowerdir=$BASE/lower,upperdir=$BASE/upper,workdir=$BASE/work
> for d in boot dev proc run sys ; do mount --bind /$d $OVERLAY/$d ; done
> mkdir $ROOT_MOUNT ; mount --bind / $ROOT_MOUNT

systemd/pid1 will mount proc/run/sys/dev/… on its own after "systemctl
switch-root", no need to do that yourself.

> ----
> This all seems to work. The overlayfs prevents writes to persistent media
> in usual places. Have a path to write to persistent media. The special
> directories (that do not work from mounts to lower on an overlayfs) work as
> expected.
>
> What I do not have as yet is a means to get the overlayfs root mounted
> early in boot.
> 1.  Does not look like /etc/fstab can create the overlayfs.
> 2.  Tried the "systemd.volatile=yes" kernel command line, w/o luck.
> Substantial commits between 239 and 245 around this.
> 3.  See systemd "pivot_root" and "switch_root", but not yet puzzled out
> usage.

Look what src/volatile-root/volatile-root.c in current systemd
versions is doing. It's relatively straight-forward. You can do
something similar with a shell script.

Lennart

--
Lennart Poettering, Berlin