[systemd-devel] Direct systemd-journald event-logs tranmssion to Zabbix\Cacti

[Lennart Poettering]

On Sat, 24.10.15 19:01, Mikhail Kasimov (mikhail.kasimov at gmail.com) wrote:

> 1) Non-systemd:
> +---------+    +----------------+                  +-------------+
> | System  | -->|(r-)syslog(-ng) | --> {UDP 514} -->|Log-collector|
> +---------+    +----------------+                  +-------------+
> 
> 2) Systemd-based Linux:
> +---------+    +----------------+                  +-------------+
> | System  | -->|(r-)syslog(-ng) | --> {UDP 514} -->|Log-collector|
> +---------+    +----------------+                  +-------------+
>     |                  A
>     |                  |
>     |                  | {/etc/systemd/journald.conf:
> Store=volatile,ForwardToSyslog=yes}
>     |                  |
>     |          +-----------------+               +-------------+
>     ---------> |systemd-journald | --> {???} --> |Log-collector|
>                +-----------------+               +-------------+

Well, rsyslog tends to get the data of the journal on its own these
days, ForwardToSyslog= is not used anymore and now defaults to
off. For rsyslog this has the benefit that it can get access to the
structured log data and not just the baseline that syslog offers. It
also gets access to logs that happened during early system-boot that way.

> If we want to send systemd-journald logs to remote log-collector system,
> we have to set up forwarding to (r-)syslog(-ng) service.
> 
> Systemd's native tools -- systemd-journal-gatewayd\-upload\-remote --
> are designed to transmit logs between only homogeneous (systemd-based)
> systems.

Well, the protocol is just HTTP(s), and as payload there are multiple
formats supported including JSON.

It'd would be easy for external services to simply speak HTTP and get
the data out of there.

We didn't really come up with a new protocol there, we just decided
not to use UDP syslog, but something more featureful, that knows
strict ordering, retransmissition, complex datatypes, is restarable,
and so on.

> 1. Because systemd-journald has NO a documented specification on
> logs-transmission transport, admins have to install other log-system in
> parallel to systemd-journald and set up systemd-journald in right way to
> have possibility to send event-logs to log-collector system (Redundancy,
> because two event-log services are in use).

Well, if you want BSD syslog, then install some form of BSD syslog
server, like syslog-ng or rsyslog.

If you want HTTP then systemd-journal-gateway works fine.

> 2. To avoid the redundancy mentioned above, admins can stop and disable
> systemd-journald.socket and systemd-journald.service and use only
> (r-)syslog(-ng) service (one service in use).

You cannot really do that. The journal is not optional. Also you will
lose early-boot stuff, and everything collected from service
stdout/stderr/... that way. 

> 3. If systemd-journald has a documented specification (RFC) on
> logs-transmission transport, admins will have pretty good choice
> (technically and/or ideologically) to use systemd-journald and\or
> (r-)syslog(-ng) at the same time or separately to send event-logs to
> Zabbix\Cacti\other log-collector.

Well, you appear to insist in RFCs a lot. Turns out BSD syslog is
particularly bad on those actually, as during most of its history it
had none. The syslog RFCs were written much much later only
documenting what has been seen in the wild, and that differs
widely. The newest iteration of the BSD syslog RFC is a different
beast even, as it documents structured stuff that pretty much nobody actually
implements...

OTOH, we implement HTTPS and JSON and stuff. Those have pretty good
RFCs actually. Sure, they aren't traditionally used for logging, but
then again, they are much more suitable for it, and fairly well
supported in all kinds of software frameworks.

Lennart

-- 
Lennart Poettering, Red Hat