[systemd-devel] Stateless system - Centos8 and overlayfs on root

[Simon McVittie]

On Fri, 27 Mar 2020 at 10:17:37 -0700, Preston L. Bannister wrote:
> Updates would be accomplished by booting from the original boot menu entry.
> (This is slightly complicated by the fact the target systems' computers do not
> have a console - but figure I can script altering the default boot.)

I think you might be better off with something based on the idea of
building an immutable "golden image" elsewhere, and deploying it to the
stateless machines using a filesystem-image-based "A/B" update strategy
(like casync and Android) or a filesystem-tree-based atomic update
strategy (like libostree, as used in Fedora Silverblue and Endless).
Either way, the big ideas are:

- build the "golden image" in CI infrastructure somewhere else
- have enough space for two complete OSs
- whichever one you are booted into (let's say it's "A"), overwrite the
  other one (let's say "B") with the new OS
- reboot into "B", but just once (have a clever bootloader)
- if "B" fails to boot, recover by going back to "A"
  - you hope that a later upgrade will fix the regression and overwrite
    "B" again, this time with a better version
- if "B" boots OK, keep using it (tell the bootloader to boot it every
  time from now on); next time you upgrade, swap the roles of "A" and "B"

Most people who do that will be using an overlay-like arrangement to
store whatever state they want to transfer between A and B, such as /etc
and /home - but in your case you don't want to store any state at all.

    smcv