[systemd-devel] Boot ordering

[Kai Krakow]

Reindl Harald <h.reindl at thelounge.net> schrieb:

> Am 19.03.2015 um 22:04 schrieb Kai Krakow:
>> Christoph Pleger <Christoph.Pleger at cs.tu-dortmund.de> schrieb:
>>
>>> I am experimenting a little with systemd and trying to define a new
>>> "intermediate" runlevel, a runlevel between basic.target and
>>> multi-user.target. This means that I want the services which are
>>> required by my new runlevel to be started after all services from
>>> basic.target have been started and to be finished before any service
>>> from multi-user.target is started.
>>
>> I think there is still this sysvinit-related misconception that systemd
>> target equal sysvinit runlevels.
>>
>> Systemd uses automatic ordering of units through socket activation and
>> explicitly defined dependencies (which BTW most of the time are not
>> needed). Thus, a target just defines which set of services you want to
>> run asap. Multiple targets can become active in parallel, and systemd
>> will try to reach them in parallel. All dependencies will be thrown into
>> the same soup and systemd works its way through it
> 
> that is all fine but given that you don't know when a service is ready
> to accept connections and the whole world don't turn around systemd it
> is a legit need to control ordering sometimes strictly

Well I don't see where the conflict is. Double-forking services should 
probably just not return to the caller until they are ready to accept 
connections. This is why you should design your service units for systemd to 
not fork but start the process in foreground (most services have a cmdline 
option for that since years). That is a clear signal for systemd that the 
service is ready (other signals are the appearence of a PID file for forking 
processes, but that is not recommended). Forking or double forking only gets 
in the way for that and is a design from sysvinit days to actually speedup 
boot (which you state should not be the priority, and I agree with that) 
and/or decouple the console so the process does not become killed by the HUP 
signal, or can setuid/setgid. If you don't want to trust some of those 
security measures to systemd (or start-stop-daemon in openrc, or whatever) 
then it is probably "your" task to ensure that the process is ready when the 
parent returns to the caller after forking. This is something often done 
"wrong" even for sysvinit - otherwise I cannot explain why we need "sleep 5" 
or "while try-something; do sleep 1; done" in init scripts. And yes, I added 
such fixes myself to init scripts in the past. And not needing it with 
systemd has nothing to do with performance but with reliability, altough I 
appreciate the performance benefit of it and it also looks much cleaner 
without "hacks". This goes along with clever tricks of init script 
maintainers who added "&" to the process spawning "to speed up things" which 
I had to remove sometimes to get a much slower but at least reliable service 
startup.

> boot performance is not the only important thing - better said on
> servers it don't matter that much given you don't boot the machine all
> day long - relieable boot in every context is much more important while
> claiming everybody does soemthing wrong and systemd is right with it's
> assumptions don't help much

You cannot simply say "boot performance is not important for servers". I 
actually would easier accept "boot performance is not important for 
desktops" because we have hibernate/suspend/sleep - desktops don't really 
need to reboot except for updates.

With modern servers which offer UEFI boot, POST time can be reduced to 
seconds when we had minutes sometimes in legacy BIOS. So, with the arrival 
of such servers we also have the need for fast boots - because service 
downtime can be a cost factor. In that regard, I cannot agree with you 
although I think that your argument is true for some, maybe most, scenarios.

But let's also elaborate on "reliable boot": This is, what systemd actually 
offers and even can guarantee, in a much better way traditional sysvinit 
could ever do. It provides well defined targets telling me if a set of 
services is up (sysvinit doesn't). It provides supervising, and it provides 
socket activation with automatic blocking. For services designed to support 
systemd (and I don't think that is such a big deal for most services), you 
could even restart services without service interruption because socket 
activation can guaratee uninterrupted (though delayed) service provision. 
You may argue there are supervising managers like supervise, runit, etc. But 
first most of them cannot use socket activation and thus do not give the 
same guarantees, and if they do, they need to use the same or very similar 
modifications to your service as they need to use for supporting systemd. 
And after all one can still argue that using facilities like supervise is no 
longer traditional sysvinit - so where's the point?

Noone claims that everyone is doing it wrong, neither that systemd is the 
only one doing it right. Everything is fine with traditional sysvinit, with 
its double-forking, early returns before a service is ready, unreliable 
dependency startup (if something goes wrong it really goes wrong), more or 
less random sleep commands to solve that very problem, artificial runlevels 
to unreliably "ensure" a set of services running... Well, it is just not the 
same quality of what systemd could do better (and in a different way, yes I 
heard you, that can be a painful learning curve). And what stops you from 
setting up a minimal systemd runlevel/target, and finally in the end start a 
service unit which runs all your nifty sysvinit'ish initializations from a 
traditional sysvinit script?

If run sysvinit-style servers for 15+ years now - and sysvinit startup _IS_ 
unreliable. Services sometimes randomly fail to start or dependent services 
timeout for no obvious reason... Most times it was fixed by inserting sleeps 
before or after service spawning, or inserting some unexpected dependencies 
or moving services between different runlevels. This has ended with systemd. 
If it works, it reliably works throughout the complete dependency chain or 
at least can clearly tell me what went wrong through the journal without 
mixing up error messages and console output of the boot process (which is a 
PITA on headless systems anyways) or either distributing logs across various 
different text log files or putting everything in one monolithic log file 
without serious tools to filter and sort that stuff (c'mon, grep, really?). 
And it also clearly knows when and why a service died while sysvinit often 
happily announced it still running and even refused to restart it because 
the PID file was still there (PID files are broken by design anyways). 
Sorry, I cannot accept that as "reliable". It even is not reliable if you 
follow the rule "if it works never touch it". BTW: You need security updates 
one time or another.

I can agree, however, with the fact that systemd is the only init system 
currently offering those features and that this is taking away freedom of 
choice in a way. But every service that is systemd compliant is still able 
to run without systemd, from traditional sysvinit - at least I yet didn't 
see anything that wouldn't. Integrating dbus, udev, or other things (and 
also the vice-versa: integrating systemd as runtime dependency into other 
ecosystems like Gnome) should clearly not become subject of this discussion, 
that's a whole different point.

But on the other hand, I think it is not that bad having systemd around and 
its "wide acceptance": For (probably) the first time in a long row of years 
Linux sees some reliable, consistent API between different distributions (I 
think its diversity is equally one of its greatest strengths and at the same 
time its biggest opponent). But this "milestone" now is IMHO a good thing 
because developers can concentrate on creating great tools building on it 
("it" being the API, I clearly don't want to say "systemd" here) instead of 
solving always the same problems over and over again. And let's just say: 
Why should systemd take the sole role of implementing this API? I think 
other, yet to come, init systems could also implement it. The design behind 
systemd and its API is not that bad, I'm sure you could agree with it. If it 
has to be systemd or an alternate implementation in the end, that's a matter 
of taste.

We wouldn't be here if new technology had always been rejected. We are here 
because new technology was adopted, by different people, in different ways, 
and most successfully with common standards, in the last 150 years and even 
beyond.

I don't want to get out of scope here. But I think the problem is that 
humans naturally refuse to accept change. Even more if they weren't asked. 
And I can agree with that: it can feel that systemd was forced on us. But I 
cannot count myself to those people because I eagerly wanted to try this 
technology and it worked for me, and with Gentoo I still had choice to use 
openrc. I remember when dbus "was forced on us", I hated it because the 
migration in my DE didn't work well for me. Nothing seemed to work reliably 
for a long time. But now it's everywhere and integral part of every distro. 
Similar to udev though I never had a problem with it - but in the end it 
looked very useful to me although I had some hickups with device naming etc.

The point is: Let's just find out why the "intuitive" way to solve the OPs 
problem doesn't work out and find the right solution. Let's face it: Trying 
to use targets as sysvinit runlevels equivalent is obviously not the working 
way although it looks promising and intuitive at first (I'd probably tried 
the same first). That kind of problem (in a common sense) is something one 
will face over and over again, unrelated to systemd. What is so special, so 
different, when it's related to systemd?

-- 
Replies to list only preferred.