[systemd-devel] Antw: Re: /etc/fstab obsolete?

[Ulrich Windl]

>>> Lennart Poettering <lennart at poettering.net> schrieb am 28.08.2019 um 14:41
in
Nachricht <20190828124149.GA8141 at gardel-login>:
> On Mi, 28.08.19 09:33, Ulrich Windl (Ulrich.Windl at rz.uni‑regensburg.de)
wrote:
> 
>> Hi!
>>
>> systemd in SLES 12 is causing endless frustration here:
>>
>> Yesterday I was migrating some filesystems to a new device (multipath, 
> MD‑RAID, LVM, filesystem, mountpoints, etc.), updating /etc/fstab and other

> files as needed.
>> After migration was successful, I also cleaned up the now obsolete
resources 
> (multipath, MD‑RAID, filesystem, mountpoints, etc.)
>> Everything looked OK...
>>
>> But some time later the application was stopped, as the new
>> filesystems were unmounted by systemd (even though active processes
>> were using it) WITHOUT giving a reason for "Stopped target Local
>> File Systems" in syslog. Instead systemd tried to mount the
>> filesystems that had been removed from /etc/fstab!
> 
> You need to issue "systemctl daemon‑reload" when updating this file,
> systemd only takes notice of changed configuration files when you
> issue that.
> 
> This is documented widely, for example here:
> 
> https://www.freedesktop.org/wiki/Software/systemd/Incompatibilities/ 
> 
> Moreover, well‑meaning distributions would most likely include a
> comment about this in /etc/fstab to make this discoverable easily by
> the admin. On Fedora you'll find this comment at the top of the file
> for example:
> 
> <snip>
> # After editing this file, run 'systemctl daemon‑reload' to update systemd
> # units generated from this file.
> </snipe>
> 

Unfortunately it's only half of the truth: Even if you do not modify
/etc/fstab, but do manual mounts or unmounts, systemd will override your
actions whenever it feels like it.

So the racy burden is now for root: You'll have to manipulate mounts and
/etc/fstab synchronously, faster than systemd can react.
I very much prefer what we had before: Commands are executed when you tell
them to be executed.

> systemd does not automatically reload configuration files when they
> change since that is inherently racy: systemd reads a larger number of
> unit files and other configuration files and simply by watching file
> changes with inotify we cannot know when the right time is to reload
> configuration: we might end up seeing some of the old
> unit/configuration files and some of the new unit/configuration files,
> as they are manipulated if we reload at the wrong time. For example,

You just have to separate knowing the current fstab from adjusting the current
state to the config.


> consider the case where you rename a unit file: there is likely a
> timeframe where enumerating all unit files might see just the old
> name, and one where it would see just the new name. It might also
> happen that enumeration sees the file both under the old and the new
> name, or under neither (simply because file enumeration is a long
> process). In all these cases we'd end up reading a borked
> configuration, hence we just don't do that.

Maybe implement "transactions".

> 
> It is expected for the admin to issue "systemctl daemon‑reload"
> explicitly at the correct time, i.e. when a set of changes to
> configuration and unit files is complete and before the next set of
> changes is begun. Only the admin really knows when that is. (I mean,
> the alternative would be if we'd have a transactional file system on
> Linux, but we do not, and I don#t see that coming either.)

The problem is the triggers: DIscoverying some changed SAN device seems to
trigger re-evaluating the mounts. However if you are migrating filesystems from
one SAN device to another, you will always have temporarily inconsistent states
(like mounting a filesystem to a different mount point to make a consistent
copy). Maybe we need a "systemd suspend" and "systemd wakeup" to be able to
work with the system. Admin-made transactions you could say.

> 
> (Another example where things might go very wrong if we'd start
> reading configuration files at the wrong time: consider RPM
> removing/updating/installing a bunch of unit files in one go: if we'd

But that already happens now: Reading a config file at the wrong time is just
as bad as continuing with obsolete config information!

> wake‑up whenever even one file changes we'd very likely wake up at the
> wrong time and see a file removed with its replacement not there yet,
> or the opposite. So if we did that and you update a bunch of RPMs at
> once you'd basically get the guarantee that you will hit this case.)

Maybe systemd could go into suspend mode automatically once a config file was
changed until the operator confirms to re-read the new configuration.


> 
>> So I tried to "start local filesystems" after realizing the problem
>> this morning. Then disaster (named "systemd") strikes back:
> 
>> It tried to mount the old filesystems that do no longer exist (and
>> are no longer present in /etc/fstab), resulting in a "dependency
>> failed", and in turn it transitioned a fully running server from
>> multi‑user mode to emergency mode, shutting down all services,
>> network, etc.
>>
>> That is why I hate systemd!
> 
> So, there are changes between systemd and pre‑systemd, non trivial
> ones, even. They are documented though, and generally well
> communicated. Yes, there's a learning curve, and yes you have to read
> some docs if you have a non‑trivial setup you want to convert to
> systemd. But I am not sure why you unload this on our doorstep. There
> are technical reasons for these choices, documented ones even. If you

Actually there are companies and software around that try to force you how to
do things instead of doing things the user wants it. Of course such systems are
documented.

I really don't understand what "unload this on our doorstep" means: Bring a
common problem to your attention, while you consider any problems to be the
user's fault?


> convert a machine from pre‑systemd systems to systemd, please
> read the docs thoroughly, and understand the incompatibilities that
> there are. We write these docs because we know how important this is,

Actually the docs refer to each other in a cyclic way, leaving the impression
that the design of systemd is not very clear, but evolving. There are 145
manual pages directly referring to "systemd". Init probably had less than four,
maybe even one. Systemd is just too complex.


> not so that you then ignore all we did and blame us anyway. And no,

Maybe add a "systemd common pitfalls" document to start with. I'm serious. I
guess you'll learn a lot from writing such a document.

> doing a live‑update between major OS releases of production
> infrastructure without reading the docs is a very very bad idea, but
> not one we came up with, did we?

You misunderstood: We are retiring an 8 year old SAN storage system on a
server that isn't allowed to go offline. I did that with different brands of
UNIX in the past, but nothing was as counter-productive as systemd.

> 
> I mean, short of actually *being* sysvinit there's literally nothing
> we could do to make you happy, can we?

Maybe add another document "Reasons to be different from init" or "Why init
and systemd cannot co-exist".

> 
> Anyway, please vent your frustrations elsewhere, this is not helpful
> nor new.

So why did you write systemd? Not frustrated with init?

> 
> And: RTFM!
> 


Regards,
Ulrich

> Lennart
> 
> ‑‑
> Lennart Poettering, Berlin