[systemd-devel] How to factory reset?

[Tobias Hunger]

On Tue, Mar 10, 2015 at 6:38 PM, Lennart Poettering
<lennart at poettering.net> wrote:
> On Tue, 10.03.15 18:13, Tobias Hunger (tobias.hunger at gmail.com) wrote:
>
>> > So you want not just factory reset, but actually a stateless system,
>> > where every single boot is basically a factory reset?
>>
>> Yes, but I do have a state that I want to be applied by default at
>> all times.
>
> Well, you want the "factory defaults" to be applied when the machine
> comes up, right?

Kind of:-) I have a somewhat strange mix of ideas taken from docker
and "Revisiting How We Put Together Linux Systems":-)

Basically I have a system a bit like docker that creates images for
all the machines I use each night:, very similar to what docker does:
You have a file, that defines some base image (which may be empty) and
allows you to further customize that image, creating a new image.

Unlike docker I create snapshots in a btrfs filesystem, mostly
following the ideas you brought forward in the "Revisiting" blog. The
end result is a "root:system:vendor:arch" and a
"usr:vendor:arch:timestamp" snapshot per physical or virtual machine I
run.

So initial install of a system is really simple: Partition the HDD,
set up a btrfs volume, copy root:s:v:a and a current usr:v:a:t over
the network, run a script that updates gummiboot based on usr:v:a:t
subvolumes found and reboot. Updates are even simpler: Copy a new
usr:v:a:t over, update gummiboot using the same script, reboot.

Currently /etc is a symlink to /usr/someplace/etc. That works well,
but has the downside that -- since /usr is mounted ro -- I can not
just experiment with different configurations anymore. Well, not
without remounting /usr rw that is.

I think I can do better with all those cool ideas like factory reset
and stateless system and such floating around:-)

> Hmm, currently we focussed on two models:
>
> a) fully "volatile", meaning / as tmpfs, with /usr mounted from
>    physical media
>
> b) with only "volatile" state, but persitent configuration, meaning /
>    mounted from physical media, and /var as tmpfs.
>
> Your model appears to be different from that. You actually want /var
> from from physical media, but /etc from tmpfs? That would be kinda the
> opposite of b)...

Yes, that is what I want.

I have a fully configured system (with that configuration stored in
/usr/someplace/etc), I want that system to reset to that configuration
regularly (in my case each time it boots), but keep the state.

> Do you actually want all of /var mounted of physical media? If you are
> interested in just logging, maybe just adding a normal mount for
> /var/log/ should suffice, leaving the rest of /var on tmpfs?

Most of the data is in /var/log, but depending on services configured
other directories in /var might be interesting, too.

I might very well decide to deploy a mail server into a VM using my
automated setup. I can then boot that and tweak the configuration,
restarting the service as necessary. If I screw up then I want a
reboot to take me back to something that works. If I improve on the
configuration I will just update my image definition and have the
change as the default the next time I update.

I definitely do not want to loose mails by rebooting the production VM
into that updated image, so the state needs to stick around.

This is probably not a very common use-case, I admit that.

> Well, factory resets are supposed to be something that gets you back
> into a defined state if you fucked up your system. In such a case it
> might not be possible to boot up anymore to reset the state... Hence
> having this on the kernel cmdline is kinda a necessity to make this
> useful in real-life...

Hmmm. Right. I assume you are going to nuking /etc without any further
user interaction, so my worry is probably unfounded.

>> Having --volatile=/path/to/usr/directory would be nice to have for the
>> experiments I do right now. I guess that is not so very common that it
>> makes sense to consider sending in a patch for that.
>
> Hmm, what precisely are you suggesting this would do?

Have / of the machine be a tmpfs and mount /path/to/usr/ into as /usr
there. Forget about this, I do not think that would be widely useful
and as script fixes me up nicely:-)

> presets and machined ID are applied by PID 1, before it begins with
> starting any units, hence *really* early on. Note though that actually
> /etc/machine-id is used as flag for "is /etc empty". If the file
> exists it is assumed that /etc is provisioned properly. If it is
> missing PID 1 will generate the machiend id and apply presets.

An OS installer could put the machine-id into /usr just fine (and so
can I) and systemd could just get it from there if available before
generating a new one.

It would be nice if the machine-id did not change during to a factory
reset: It is used in the logs and changing it will basically make
historical log data much harder to analyze. IIRC networkd also uses it
to generate persistent MAC addresses for containers and such.

So far this seems to be the only critical piece of information that
can not get restored via tmpfiles.d.

> Note though that some services like ldconfig.service also want to
> write to /etc, to generate some files, if they are missing... If you
> want to do something before that you have to order those units
> explicitly before each one of them.

I guess I will not have to worry about ldconfig.service. That is more
of a cache than configuration, isen't it?

> Note though that much like /usr, /etc is something that we assume is
> premounted when systemd is started, and where we do not support
> mounting it after systemd began its work. I mean, /etc is usually
> where moutns are configured, but if youw ant to mount someting on
> /etc, then how is that to be found?

I was well aware of /usr being required to be available when
transitioning out of initrd. I am a bit surprised about the
requirement that /etc needs to be (partially) populated as well.

Best Regards,
Tobias