[systemd-devel] A way to better integrate rsyslog into systemd?

[Aleksei Besogonov]

On 04 Oct 2014, at 00:17, Cameron Norman <camerontnorman at gmail.com> wrote:
> El vie, 3 de oct 2014 a las 10:55 , Aleksei Besogonov <alex.besogonov at gmail.com> escribió:
>> With all the recent noise about systemd abusing its position with the way it takes over logging I’ve been thinking about a way to solve it.
>> 
>> As far as I understand the following holds:
>> - Systemd takes over /dev/log socket which is normally served by rsyslog (or other syslog daemon).
>> - That’s really required to make journald-based logging transparent and coherent for most use-cases.
>> 
>> However, it creates a problem for log-heavy applications, because of additional roundtrips between processes. So far I’ve heard people actually using LD_PRELOAD tricks to hack around applications opening the /dev/log file inside the syslog(2). As far as I understand, it’s also not really configurable - the '/dev/log’ string is hardcoded into various libcs (e.g.: http://git.musl-libc.org/cgit/musl/tree/src/misc/syslog.c). Recent versions of rsyslog can directly read journald files. But that’s still suboptimal solution, because it introduces an unnecessary layer.
>> 
>> Namespacing each daemon to provide its own /dev tree with custom /dev/log sockets is possible, but impractical.
>> 
>> So I propose the following solution:
>> 1) Add an option to systemd units to allow passing opened /dev/log sockets to rsyslog (using the usual SOL_SOCKET mechanism).
> This seems annoyingly roundabout. A more straightforward method would be to just have rsyslog listen on /dev/log. Obviously that is the original method, and journald changed it, and there are reasons for that. I am pretty sure one of the main selling points of journald is the extreme catch all nature: from syslog messages, to daemon stdout/in, to X.Org logs, etc. You can easily access those logs from one place and with one set of tools.
As I understand, systemd developers indicated that they don’t want to add this option. Not in the least because affects the whole system, not just one service.

> If the authors wanted syslog messages to not be intercepted by journald, I do not think they would have had journald listen on /dev/log.
Journald is desirable and it can forward log messages to rsyslog as it is. 

However, one of our clients raised an a question about forwarding overhead. It turns out that it’s definitely non-trivial and can in some cases cause significant slowdowns during log-heavy events. Think about 400Mb/s of logs - most of them are filtered out by rsyslog locally and some are transmitted over the network for permanent storage.

So they would like to turn off the journald intercepting /dev/log, but only for specific services. I think that socket forwarding seems to be the easiest way to implement it. I’m going to actually do it and send patches for approval, if systemd developers don't object in principle or unless somebody proposes a better alternative.

> Can it not simply recognize it is listening on /run/systemd/journal/syslog and skip any kind of extra steps? Or would that cause sec issues (e.g. easier to overflow the logs by faking the PID / user then writing to the private socket directly)?
That also. And it’ll still be nice to be as transparent as possible, if required.