[systemd-devel] systemd-logger and external syslog daemon

[Mike Kazantsev]

Good day,


I've recently deployed systemd on a machine that uses some syslog
monitoring software and the software went nuts because messages from
systemd logger were inconsistent with other logging - they all look
like this:

  kernel.warning kernel[-]: process[pid]: message contents

Thus, regardless of what systemd.exec(5) states, SyslogFacility= and
SyslogLevel= is irrelevant - to a syslog daemon they will be passed as
"kernel.warning".

Naturally, the reason is "systemd-kmsg-syslogd" daemon, which spawns
early on boot from syslog.socket and creates /dev/log, which is
accessed by systemd-logger long before external syslog daemon starts.
Syslog daemon (rsyslog in my case) just grabs these from kmsg.

After that, substituting /dev/log for another socket (like
rsyslog.socket seem to do) or just removing it doesn't seem to
matter - systemd-logger will keep feeding messages to dmesg via
kmsg-syslogd.


I read Kay Sievers's earlier post in this list on the subject, which
states "Syslog services have been patched to be able to seemlessly take
over an already opened /dev/log. rsyslog is upstream already...".

Yet I've tried latest scm version of rsyslog and it doesn't seem to
"take over an already opened /dev/log" if started via upstream-shipped
rsyslog.socket unit (which just uses "ListenDatagram=/dev/log") line.

So I wonder, was Kay wrong about that functionality?
Or maybe some other actions are necessary to enable the mentioned
behavior?


Just stopping systemd-kmsg-syslogd produces "endpoint not connected"
errors in dmesg from systemd-logger:

  systemd-logger[355]: Got error on stream: "Connection refused"
  systemd-logger[355]: Got error on stream: "Transport endpoint is not connected"

New /dev/log meanwhile seems sperfectly fine, lsof shows that it belongs
to an rsyslogd process and other software seem to use it w/o problem,
so I have no idea where that "Connection refused" comes from.

"strace -f" of a systemd-logger (latest systemd git) process doesn't
show it trying to connect anywhere at all, just two sendmsg's with the
message, resuling in ESRCH and ENOTCONN errnos and one writev with the
second error message.

Restarting systemd-logger seem to be an option, although in one case
it seemed to result in an emergency shutdown of machine, yet I doubt
that "ExecStartPost=/bin/systemctl restart systemd-logger.service" is
the intended solution here.


I was able fix (or maybe work around) the issue by adding
"DefaultDependencies=false" to rsyslog.socket unit and masking original
syslog.socket.

That way kmsg-syslogd doesn't get started at all and systemd-logger
uses rsyslog.socket-provided /dev/log as intended.

There seem to be some problems (timeout) on shutdown though (which I
need to investigate, but pretty sure something hangs on /dev/log), so
this doesn't seem to be a perfect solution either.

How the kmsg-syslogd->rsyslog migration (for systemd output) is supposed
to work?


-- 
Mike Kazantsev // fraggod.net