[systemd-devel] Read-only /etc, machine-id with an overlay - journald failing

[Andreas Kempe]

On Tue, Mar 31, 2020 at 03:52:44PM +0200, Lennart Poettering wrote:
> On Mi, 26.02.20 09:44, Andreas Kempe (andreas.kempe at actia.se) wrote:
> 
> > Hello everyone,
> >
> > I'm working in a project with an embedded Linux system based on
> > Openembedded using Systemd version 241 as our init process. We're
> > using a read-only /etc. To facilitate development, we want to use a
> > writeable overlay on /etc, but we ran into an issue.
> >
> > When we start, Systemd detects that there is no machine-id file
> > present in /etc so it generates and mounts a /etc/machine-id. When our
> > mount unit then applies the overlay on /etc, it hides the mounted
> > file. Journald later fails to start because /etc/machine-id isn't
> > visible through the overlay.
> 
> So when you swap out /etc you really should so so atomically, and
> ensure that at no point in time stuff (i.e. mounts) go missing while
> you shift things around. In particular not the machine-id.
> 

Yes, this is sound advice and of course the reasonable way to handling
moving file systems around. We weren't counting on having anything
extra mounted in /etc so it took us a bit by surprise.

> Hence, the code that replaces /etc with an overlayfs, why doesn't it
> bind mount /etc/machine-id into the new overlayfs before moving it
> into place? i.e. first mount the overlayfs to some tmpdir, bind mount
> /etc/machine-id into it, then use mount --move to move the fully
> prepped overlayfs along with all submounts to /etc. That way noone
> will notice anything changing.
> 

The simple answer for not doing the above is that we didn't think of
the idea. Personally, I don't have very much experience with overlays
and I am used to simply mounting things at boot where they should be
one thing at a time.

One learns something new every day. Thank you for the advice! :)

Cordially,
Andreas Kempe