[systemd-devel] A way to better integrate rsyslog into systemd?

["Jóhann B. Guðmundsson"]

On 10/04/2014 05:55 AM, Aleksei Besogonov wrote:
> With all the recent noise about systemd abusing its position with the 
> way it takes over logging I’ve been thinking about a way to solve it.
>
> As far as I understand the following holds:
> - Systemd takes over /dev/log socket which is normally served by 
> rsyslog (or other syslog daemon).
> - That’s really required to make journald-based logging transparent 
> and coherent for most use-cases.
>
> However, it creates a problem for log-heavy applications, because of 
> additional roundtrips between processes. So far I’ve heard people 
> actually using LD_PRELOAD tricks to hack around applications opening 
> the /dev/log file inside the syslog(2). As far as I understand, it’s 
> also not really configurable - the '/dev/log’ string is hardcoded into 
> various libcs (e.g.: 
> http://git.musl-libc.org/cgit/musl/tree/src/misc/syslog.c). Recent 
> versions of rsyslog can directly read journald files. But that’s still 
> suboptimal solution, because it introduces an unnecessary layer.
>
> Namespacing each daemon to provide its own /dev tree with custom 
> /dev/log sockets is possible, but impractical.
>
> So I propose the following solution:
> 1) Add an option to systemd units to allow passing opened /dev/log 
> sockets to rsyslog (using the usual SOL_SOCKET mechanism).
> 2) Add the corresponding functionality to rsyslog. It should listen on 
> a special socket (perhaps /run/rsyslog/socket_server ?) and treat all 
> the incoming sockets as if they were accepted from /dev/log.
>
> It would also solve the problems with rsyslog using its own 
> SCM_CREDENTIALS lookups. 

Tying the solution to rsyslog or syslog-ng is not something that should 
be done but rather fix the usability.

Based on what I have observed thus far people uses of sending journal 
logs to centralized syslog server of any kind boils down to...

"journalctl -o <format> -f | ncat <protocol><host><port>"
"journalctl -o <format> -f | <own written python tool/filter who then 
sends the logs> <protocol><host><port>"

As well as adding somekind of token in front of the log entries ( 
cloud/container specific ) as in
"ncat --ssl --ssl-verify <host> <port> < <(awk -v token=token_here '{ 
print token, $0; fflush(); }' < <(journalctl -o short -f))"

Embedded uses netconsole as in

systemd.log_target=kmsg 
netconsole=[src-port]@[src-ip]/[],[tgt-port]@/[tgt-macaddr]

So the solution which combines these common denominators thus should 
eventually make everyone happy is integrate something like netconsole...

Then add following lines to journald.conf

ForwardToHost=yes
ForwardToHostIP=<Destination IP>
ForwardToHostPort=<Destination Port>
ForwardToHostFormat=<short,json etc >
ForwardToHostLogToken=<insert token who get appended before each log 
line here probably should default to machineid>

If someone does provide patches that does just that or similar I think 
this outspoken shortcoming of journal can be finally put to rest and 
people be happy.

JBG