[systemd-devel] A way to better integrate rsyslog into systemd?

[Lennart Poettering]

On Fri, 03.10.14 22:55, Aleksei Besogonov (alex.besogonov at gmail.com) wrote:

> With all the recent noise about systemd abusing its position with
> the way it takes over logging I’ve been thinking about a way to
> solve it.
> 
> As far as I understand the following holds:
>
> - Systemd takes over /dev/log socket which is normally served by
>   rsyslog (or other syslog daemon).
>
> - That’s really required to make journald-based logging transparent
>   and coherent for most use-cases.
> 
> However, it creates a problem for log-heavy applications, because of
> additional roundtrips between processes. So far I’ve heard people

"roundtrips"? The syslog protocol doesn't really do roundtrips, it is
strictly one-way.

Do you have any data on journald not being able to keep up? I'd
certainly be interested to fix that?

Does increasing /proc/sys/net/unix/max_dgram_qlen make things better?

> actually using LD_PRELOAD tricks to hack around applications opening
> the /dev/log file inside the syslog(2). As far as I understand, it’s
> also not really configurable - the '/dev/log’ string is hardcoded
> into various libcs (e.g.:
> http://git.musl-libc.org/cgit/musl/tree/src/misc/syslog.c). Recent
> versions of rsyslog can directly read journald files. But that’s
> still suboptimal solution, because it introduces an unnecessary
> layer.

Well, there's a reason journald is where we centralize
logging. Traditionally, /dev/log is only available from late boot on,
not during early boot or even the initrd. journald actually collects
data all the time, from the earliest moment on, and then allows a
syslog implementation to pick it up. Morevoer, it actually collects
logs from a number of additional sources, including, and most
importantly from stderr/stdout of all system services. Because we
connect all services with it, from the earliest boot on, journald is
not really optional, it's a system component, written in a way that it
can run at a time the system is not initialized fully yet, with no
/var around, or anything being writable yet. You cannot just replace
it with rsyslog hence, because rsyslog cannot cover all of this.

> Namespacing each daemon to provide its own /dev tree with custom
> /dev/log sockets is possible, but impractical.

In recent versions of systemd you can just override Symlinks= in
systemd-journald-dev-log.socket and set it to the empty array, and
then add Symlinks=/dev/log in syslog.socket instead, which should
change /dev/log from being directed to the journal, to beign directed
directly to your syslog instance. But note again, that will lose much
of the early-boot loggind that way. Also "systemctl status" of course
won't show you any logs anymore (or at least incomplete ones).

> So I propose the following solution:
> 1) Add an option to systemd units to allow passing opened /dev/log
> sockets to rsyslog (using the usual SOL_SOCKET mechanism).

No, while bypassing journald is certainly possible, I am pretty sure
this is nothing we should explicitly supported, due to the issues
pointed out above. I'd much rather figure out why people actually
would want to do that, and simply fix those issues.

> 2) Add the corresponding functionality to rsyslog. It should listen
> on a special socket (perhaps /run/rsyslog/socket_server ?) and treat
> all the incoming sockets as if they were accepted from /dev/log.
> 
> It would also solve the problems with rsyslog using its own SCM_CREDENTIALS lookups.

Hmm?

Lennart

-- 
Lennart Poettering, Red Hat