[systemd-devel] Direct systemd-journald event-logs tranmssion to Zabbix\Cacti

[Mikhail Kasimov]

Hello Lennart! The main question is not about why rsyslog is
better/worst compared to systemd-journald. The main question is how to
transmit journald event-log to centralized log-server (Zabbix\Cacti\so
on) directly, without any additional assistant layers (rsyslog in this
case). JSON, HTTP(S) etc. -- that's great, but HTTP(S) is actually used
for web-surfing. And it often procced with port-redirections (on Squid
for example), shaping the speed depending on web-content and so on.

"It'd would be easy for external services to simply speak HTTP and get
the data out of there." (с) - yes, but there're many factors (traffic
shaping, other firewall ruling etc.) that potentially can affect on
systemd-journald logs transmission. So, the systemd-journald's dedicated
documented transports (open and secure) will be good idea.

This will make the setting up the logs transmission  much more easy and
we will not be depended on other web-content, channel overloads, traffic
shaping and other busy-stuff.

Besides, having a dedicated documented transport will be nice according
to security standards (e.g. PCI DSS). They're very sensitive about data
tranmssion and log-servers. What should I tell to auditor - "yes, my
logs are sent on not documented proto or on web-content proto!"? :)

If we send logs between internal helpers - gatewayd/-upload/-remote
(systemd sandbox of Juche :) ) -- no problem.

But when admin needs to have an external log-collector system (according
to different security standards), and -- that vey important -- in
realtime mode, he needs dedicated documented proto (like UDP 514 for
rsyslog), that has no risks on traffic shaping and so on.


"we just decided not to use UDP syslog, but something more featureful,
that knows strict ordering, retransmissition, complex datatypes, is
restarable, and so on... The syslog RFCs were written much much later
only documenting what has been seen in the wild, and that differs
widely. The newest iteration of the BSD syslog RFC is a different
beast even, as it documents structured stuff that pretty much nobody
actually implements..."

======
I'm not against pretty good features of systemd. :) I'd only like to
have dedicated systemd-journald documented UDP\TCP port to send
event-logs _directly_ (without any additional assistant layers) to
log-collectors sensors like it exists for rsyslog on UDP 514 and
syslog-tls         6514/tcp     # Syslog over TLS  [RFC5425]
syslog-tls         6514/udp     # syslog over DTLS  [RFC6012]
syslog-tls         6514/dccp    # syslog over DTLS  [RFC6012].

With maximum setting up like: iptables -p udp --dport $journald_port -j
REDIRECT --to-port 514.

26.10.2015 13:00, Lennart Poettering пишет:
> On Sat, 24.10.15 19:01, Mikhail Kasimov (mikhail.kasimov at gmail.com) wrote:
> 
>> 1) Non-systemd:
>> +---------+    +----------------+                  +-------------+
>> | System  | -->|(r-)syslog(-ng) | --> {UDP 514} -->|Log-collector|
>> +---------+    +----------------+                  +-------------+
>>
>> 2) Systemd-based Linux:
>> +---------+    +----------------+                  +-------------+
>> | System  | -->|(r-)syslog(-ng) | --> {UDP 514} -->|Log-collector|
>> +---------+    +----------------+                  +-------------+
>>     |                  A
>>     |                  |
>>     |                  | {/etc/systemd/journald.conf:
>> Store=volatile,ForwardToSyslog=yes}
>>     |                  |
>>     |          +-----------------+               +-------------+
>>     ---------> |systemd-journald | --> {???} --> |Log-collector|
>>                +-----------------+               +-------------+
> 
> Well, rsyslog tends to get the data of the journal on its own these
> days, ForwardToSyslog= is not used anymore and now defaults to
> off. For rsyslog this has the benefit that it can get access to the
> structured log data and not just the baseline that syslog offers. It
> also gets access to logs that happened during early system-boot that way.
> 
>> If we want to send systemd-journald logs to remote log-collector system,
>> we have to set up forwarding to (r-)syslog(-ng) service.
>>
>> Systemd's native tools -- systemd-journal-gatewayd\-upload\-remote --
>> are designed to transmit logs between only homogeneous (systemd-based)
>> systems.
> 
> Well, the protocol is just HTTP(s), and as payload there are multiple
> formats supported including JSON.
> 
> It'd would be easy for external services to simply speak HTTP and get
> the data out of there.
> 
> We didn't really come up with a new protocol there, we just decided
> not to use UDP syslog, but something more featureful, that knows
> strict ordering, retransmissition, complex datatypes, is restarable,
> and so on.
> 
>> 1. Because systemd-journald has NO a documented specification on
>> logs-transmission transport, admins have to install other log-system in
>> parallel to systemd-journald and set up systemd-journald in right way to
>> have possibility to send event-logs to log-collector system (Redundancy,
>> because two event-log services are in use).
> 
> Well, if you want BSD syslog, then install some form of BSD syslog
> server, like syslog-ng or rsyslog.
> 
> If you want HTTP then systemd-journal-gateway works fine.
> 
>> 2. To avoid the redundancy mentioned above, admins can stop and disable
>> systemd-journald.socket and systemd-journald.service and use only
>> (r-)syslog(-ng) service (one service in use).
> 
> You cannot really do that. The journal is not optional. Also you will
> lose early-boot stuff, and everything collected from service
> stdout/stderr/... that way. 



>> 3. If systemd-journald has a documented specification (RFC) on
>> logs-transmission transport, admins will have pretty good choice
>> (technically and/or ideologically) to use systemd-journald and\or
>> (r-)syslog(-ng) at the same time or separately to send event-logs to
>> Zabbix\Cacti\other log-collector.
> 
> Well, you appear to insist in RFCs a lot. Turns out BSD syslog is
> particularly bad on those actually, as during most of its history it
> had none. The syslog RFCs were written much much later only
> documenting what has been seen in the wild, and that differs
> widely. The newest iteration of the BSD syslog RFC is a different
> beast even, as it documents structured stuff that pretty much nobody actually
> implements...
> 
> OTOH, we implement HTTPS and JSON and stuff. Those have pretty good
> RFCs actually. Sure, they aren't traditionally used for logging, but
> then again, they are much more suitable for it, and fairly well
> supported in all kinds of software frameworks.
> 
> Lennart
>