[systemd-devel] [PATCH] Avoid reloading services when shutting down

[Michael Biebl]

2015-02-03 22:38 GMT+01:00 Lennart Poettering <lennart at poettering.net>:
> On Tue, 03.02.15 22:22, Michael Biebl (mbiebl at gmail.com) wrote:
>
>> 2015-02-03 22:10 GMT+01:00 Lennart Poettering <lennart at poettering.net>:
>> > I don't see how this would apply to non-sysv code. I mean, code that
>> > is written with systemd semantics in mind should be able to issue a
>> > service reload during any time it wants to, if it keeps the ordering
>> > issues in mind. For example, if I have a service that has
>> > DefaultDependencies=no (and hence ordered against nothing at all by
>> > default), and I want to issue systemctl reload on it, knowing that
>> > this cannot really deadlock, since there are no deps that could cause
>> > deadlocks, then i should be able to do so. With your patch you turn
>> > these reloads into NOPs...
>>
>> During shutdown (and early boot), yes. But why would you want to
>> schedule a restart or reload during shutdown?
>
> Well, we try to keep reloads out of the default codepaths, but it's
> easy to construct cases where people might want to deviate fromt the
> default codepaths.
>
>> Do you have a real-world example for that?
>
> Consider systemd-journald.service. It's a service with
> DefaultDependencies=no. We don't terminate it as part of the normal
> shutdown, we leave that to the final killing spree, so that we have
> logging for as long as possible.
>
> Now, people might want to hack something up that changes journald
> configuration to forward logs to kmsg during the last part of
> shutdown, so that they can see it in netconsole or so. SO they write
> their little service that patches journald.conf and restarts journald,
> and this is done during shutdown from a normal service's ExecStop=
> line. This normally works fine, since journald is not ordered against
> anything that matters. But it doesn't work on Debian, because on
> Debian there's no way anymore to restart journald as soon as shutdown
> commenced...
>
> While I just made this scenario up I think it's actually quite
> realistic, and I think it's a valid thing for admins to do....

Well, we could easily check if DefaultDependencies=yes in this case.
Actually, this is already what we do for the boot case. [1]

So even with your made-up example, it would work.

>> The thing is, you have to be extra careful to never, ever call a
>> restart/reload from such hook scripts. If those are triggered via
>> service or systemctl on a native or SysV script doesn't make a
>> difference.
>
> It is completely fine to enqueue restarts and reloads from hook
> scripts. However the emphasis is on *enqueue*. The issue is that you
> block on it, you shouldn't do that.
>
> On Fedora, iscsi is reloaded from an NM callout. If you ask me that's
> frickin' ugly, but it *is* supported as long as iscsi's reload is
> queued asynchronously instead of requested synchronously. In Fedora,
> the logic to make this async sits on the client side of things, it's
> not enforced by the engine in PID 1.
>
> The same really applies to Debian too...
>
>> It's simply to easy to cause a dead lock this way, and I think systemd
>> should try much harder to avoid such situations.
>
> Well, it would be great if we could solve the halting problem. But we
> can't.
>
> I mean, I am all ears regarding adding deadlock detection code. But I
> am really not convinced that breaking the engine for *everybody* just
> because *some* clients are weird is an option.

Calling it "breaking the engine for everybody" is hyperbole.

That said, do you have better ideas how we could avoid having systemd
easily being dead-locked on shutdown?
I'm all for solving it in a nicer way. But simply throwing the hands
in the air and saying, not our problem, doesn't help.
Things like that hurt our users badly. systemd should always try to
get into a usable state, i.e. to boot into state where one can log in,
or shutdown/reboot successfully.
As much as I like systemd, it's really easy to break it currently and
it would be awesome if it was more robust in such situations.


Michael


[1] http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/tree/debian/patches/Avoid-reload-and-re-start-requests-during-early-boot.patch

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?