[systemd-devel] Read-only /etc, machine-id with an overlay - journald failing

[Andreas Kempe]

On Thu, Feb 27, 2020 at 10:04:37AM +0100, Jérémy ROSEN wrote:
> Le mer. 26 févr. 2020 à 10:59, Andreas Kempe <andreas.kempe at actia.se> a
> écrit :
> 
> > Hello everyone,
> >
> > I'm working in a project with an embedded Linux system based on
> > Openembedded using Systemd version 241 as our init process. We're
> > using a read-only /etc. To facilitate development, we want to use a
> > writeable overlay on /etc, but we ran into an issue.
> >
> > When we start, Systemd detects that there is no machine-id file
> > present in /etc so it generates and mounts a /etc/machine-id. When our
> > mount unit then applies the overlay on /etc, it hides the mounted
> > file. Journald later fails to start because /etc/machine-id isn't
> > visible through the overlay.
> >
> > At this point we're considering a number of workarounds, but I thought
> > it worthwhile asking the experts before we go patching Systemd or
> > similar.
> >
> > My gut feeling is that using overlays on /etc can't be that uncommon
> > and it is likely PEBKAC on our end. Is there some canonical way of
> > doing overlays with Systemd and we're screwing things up?
> >
> > Thank you in advance for any help!
> > Cordially,
> > Andreas Kempe
>
> I had similar problems with a case of booting with the rootfs read-only and
> then becoming read-write later...
> 
> Basically systemd only checks for machine-id very early (before reading any
> config file) and does not deal well with /etc changing status...
> 

It is somewhat comforting knowing that others are seeing similar
issues. :)

> I did a complete analysis of what's going on, with a patch that improves
> the situation here : https://github.com/systemd/systemd/pull/14135
> I am not sure how to deal with it in your specific case.
> the simplest approch would be to mount your overlay in a initrd (or in a
> small script shell that is run before systemd and exec systemd as its last
> step)
> 

I was contemplating whether it could be acceptable having the same
static machine-id file pre-generated for all systems. I'm not 100% sure
what it's used for, TBH; would it be a really bad idea?

> My patch wouldn't really help in your case, but maybe you can "cheat" by
> having the underlying /etc/machine-id bein a symlink to the overlay
> directory... that could work.
> 

I had a look at your patch and as you said, it doesn't really solve
our use case. At the moment, we decided to remove the overlay from the
affected parts and simply require a new system image if one wants to
change /etc.

We were planning on having signed read-only overlays for configuration
in the future so I guess we'll have to investigate this further at a
later date.

Thank you for taking the time to respond!
Cordially,
Andreas Kempe