[systemd-devel] syslog makes impossible to enter emergency mode

[Lennart Poettering]

On Sat, 30.11.13 10:27, Andrey Borzenkov (arvidjaar at gmail.com) wrote:

> 
> В Tue, 26 Nov 2013 02:27:00 +0100
> Lennart Poettering <lennart at poettering.net> пишет:
> 
> > On Tue, 26.11.13 01:00, Lennart Poettering (lennart at poettering.net) wrote:
> > 
> > > 
> > > On Sun, 24.11.13 22:36, Andrey Borzenkov (arvidjaar at gmail.com) wrote:
> > > 
> > > > Interesting case (https://bugzilla.novell.com/show_bug.cgi?id=852021).
> > > > Systemd enters emergency due to failed mount. At the same time syslog
> > > > socket triggers syslog.service. Due to implicit Requires on
> > > > basic.target which Requires sysinit.target which conflicts with
> > > > emergency.{service,target} syslog.service tries to start basic.target
> > > > (it is not there yet ...) which apparently kills emergency shell.
> > > 
> > > This was probably introduced by
> > > 80cfe9e163b1c92f917e0a5e053b148fca790677.
> > > 
> > > I figure we should find something in the middle of
> > > OnActivationIsolate=yes and OnActivationIsolate=no. i.e. make use of
> > > the "replace-irreversible" job mode which will allow the emergency jobs
> > > to be queued without being reversible by later requests until they are
> > > finished or explicitly flushed out with "systemctl cancel".
> > > 
> > > I figure I'll replace OnActivationIsolate=yes by OnActivationMode= which
> > > takes the full range of job modes, and then turn OnActiveIsolate= into a
> > > hidden compat switch
> > 
> > Done.
> > 
> > Lennart
> > 
> 
> Does not work. Quoting man page: "creates jobs that cannot be reversed
> UNLESS THEY FINISHED or are explicitly canceled". This is exactly what
> happens here - emergency.target job is already finished, then syslog
> kicks in and tries to starts basic.target => sysinit.target which kills
> emergency service.
> 
> Nov 30 09:48:57 linux-cor4 systemd[1]: Dependency failed for Local File Systems.
> Nov 30 09:48:57 linux-cor4 systemd[1]: Triggering OnFailure= dependencies of local-fs.target.
> Nov 30 09:48:57 linux-cor4 systemd[1]: Trying to enqueue job emergency.target/start/replace-irreversibly
> ...
> Nov 30 09:48:58 linux-cor4 systemd[1]: Job emergency.service/start finished, result=done
> Nov 30 09:48:58 linux-cor4 systemd[1]: Started Emergency Shell.
> Nov 30 09:48:58 linux-cor4 systemd[1]: Starting Emergency Mode.
> Nov 30 09:48:58 linux-cor4 systemd[1]: emergency.target changed dead -> active
> Nov 30 09:48:58 linux-cor4 systemd[1]: Job emergency.target/start finished, result=done
> Nov 30 09:48:58 linux-cor4 systemd[1]: Reached target Emergency Mode.
> ...
> Nov 30 09:48:58 linux-cor4 systemd[1]: Incoming traffic on syslog.socket
> Nov 30 09:48:58 linux-cor4 systemd[1]: Trying to enqueue job rsyslog.service/start/replace
> ...
> Nov 30 09:48:58 linux-cor4 systemd[1]: Installed new job emergency.service/stop as 192
> Nov 30 09:48:58 linux-cor4 systemd[1]: Installed new job emergency.target/stop as 193
> 
> This is obvious race condition - some people reported success some -
> not. I can reliably reproduce failure :)
> 
> And it does not look like using isolate would actually change anything
> here ... we'd be in the same situation.
> 
> Short term solution is to make emergency.service conflict with
> sysylog.socket, but this potentially can be triggered by any other
> socket so it obviously does not scale.
> 
> So to repeat the question - should not *default* service dependency be
> Requisite instead of Requires?

So, there has been a long standing TODO item regarding this: use
RequisiteOverride=basic.target instead of Requires=basic.target for
services by default. This would have the result that services can be
started by the user without fulfilling basic.target, but causing it to
fail if it is pulled in recursively.

THis should fix your issue too, no?

The change to make things work like this should be fairly easy in
systemd, but so far I was always too lazy to test this in all detail to
make sure it actually works (especially since the xyzOverridable stuff is
little used and might not work entirely correct).

Can I convince you to play around with this to see if this makes things
work for you?

Lennart

-- 
Lennart Poettering, Red Hat