[systemd-devel] what does systemd do if the syslogd errs on start?

[Lennart Poettering]

On Thu, 07.07.11 08:28, Rainer Gerhards (rgerhards at gmail.com) wrote:

> 
> On Wed, Jul 6, 2011 at 4:17 PM, Lennart Poettering
> <lennart at poettering.net> wrote:
> > On Wed, 06.07.11 16:11, Rainer Gerhards (rgerhards at gmail.com) wrote:
> > Yes, if rsyslog dies, then systemd will notice it. As soon as there is
> > traffic on the /dev/log socket (which might be right-away) it will then
> > start the syslog-kmsg bridge again.
> 
> Will systemd make any attempt to restart syslogd if it failed? I think
> there is pros and cons in it:

You can configure systemd for that. Use Restart=on-failure and systemd
will restart rsyslog if it fails.

> > From a client perspective nothing of this is visible. Clients can jus
> > use syslog(), and this will go to rsyslog as long as it is started, and
> > to kmsg before rsyslog started and after it died (regardless whether
> > that was abnormally or cleanly).
> 
> Let's make sure we are on the same line. Sequece of events:
> 
> 1. systemd starts syslogd
> 2. syslogd parses config, detects errors
> 3. syslogd logs config errors via syslog()
> 4. syslogd terminates
> 
> Will that work? Note that at #3, syslogd has not yet terminated, so
> from a systemd POV it will probably look healthy.

Well, it's a game of luck whether this will work.

systemd will not try to start the kmsg log daemon as long as the rsyslog
service is not fully dead. That means that everything logged while you
are shutting down will be placed in the socket buffer but only removed
from there when a syslog (be it rsyslog or the kmsg bridge) is properly
running again. However, if you use syslog() to generate your messages
and are blocking on it and the socket buffer ran full, then you'll wait
for the log buffer to be flushed but at the same time making sure it
never gets dispatched. With other words: as soon as the socket buffer
runs full you enter a deadlock.

There's a relatively easy way to "fix" this however. While the libc
syslog() call always blocks when writing log messages it is reasonably
easy to reimplement that call and set the O_NONBLOCK flag for the socket
fd. If you do that and the log buffer is full, then your logging will
not succeed (and you'd drop it then) but the deadlock is
avoided. systemd itself does that for example: we want to keep the
number of processes we synchronously wait for at a minimum. OTOH we do
want to use syslog logging. Our way out is to use O_NONBLOCK which
means logging will work fine until the socket buffer is full in whcih
case we will automatically fall back to kmsg logging.

> Having said that, would it make sense to somehow make syslogd flag
> systemd that it had a successful startup and is now ready to process
> messages?

I don't see what we'd win by that.

Lennart

-- 
Lennart Poettering - Red Hat, Inc.