[systemd-devel] [PATCH] journald: Introduce RFC 5424 syslog

[Holger Winkelmann [TP]]

HI,

I'm feeling a bit responsible for the syslog forwarding discussion,
see comments below:

> Hmm, wasn't the last proposal we discussed to do this in an auxiliary
> daemmon, possibly in systemd-journal-upload or so, but not in
> journald?

My take away form this discussion was to have a "live" syslog remote forwarding in
journald which has limited support for the RFC5424 format(only the one log
line Format Susant describes) And doing the fancy stuff like Structured logging
with a daemon (a.k.a as systems-journal-upload) or similar.

Live forwarding acts like the usual syslog forward and is not able
to go back in time.
 
> I see two problems with journald: first of all, for security reasons I
> am conservative about making it deal with the network
> directly. Opening up such a basic daemon to the network is a something
> i'd prefer to avoid.

As I remember you mentioned love to see UDP broadcast. And a discussion on the ML,
was ending with basic Syslog forwarding with a RFC5324 log line -> Yes.

We also have some patches lurking around adding this live forwarding support,
but without UDP broadcast.

Tobias, could you send them to the ML for review please.

> The other thing is that journald runs really really early during boot,
> at a time where the network is unlikely to be up. This means that
> early boot msgs could never be delivered via syslog...

Yes, but thats the same situation with the normal remote syslog forwarding
today.

> I'd really prefer a scheme where this syslog broadcaster can be run
> relatively late at boot and where it tries to repeatedly send the
> messages, until sendmsg() actually succeeds. i.e. using the journal
> cursor logic it would not send a log message until the point where the
> previous message was delivered with a successful sendmsg(). Wth such a
> scheme all early boot msgs would be dumped on the network the moment
> the network is up.

I think this is intended behavior of a "uploader" or "gateway" which acts as a client
to the journal and (re)tries to forward messages. In a default config he
can do this i.e. from the last boot. We have such gateway including
a query API in the works [2] which forwards messages via ZMTP but also
will gain a GELF and remote syslog adapter.


> Zbigniew, do you have more ideas about this?

Zbigniew, I think the yes for basic syslog live forwarding with the minimum
RFC5424 format (without multiline structure) was coming from you? IIRRC

Holger

[2] https://github.com/travelping/zmq-journal-gatewayd/ should have ZMTP
transport as an option like GELF, Syslog, HTTP!

Sebastian, may you can push the current WIP branch as well? 

a small drawing about the Ideas:

          +------------+
          |  journald  |
          |            |
          |            |+------------+
          |            |             |
          |            |             |
          +-----+------+             |live forwarding
                |                    |
                |                    |
                | journal_api        |
                |                    |
                v                    v
          +------------+ syslog  +------------+
          |  Gateway   |-------->| SYSLOG     |
          |            |         +------------+
          |  acts      | GELF    +------------+
          |  as client |-------->| GRAYLOG2   |
          |            |         +------------+
          |            | HTTP    +------------+     +--------------+
          |            |-------->| journal_rmt+---->|  journald    |
          |            |         +------------+     |              |
          |            | ZMTP    +------------+     |              |
          |            |<------->| ZMQ (TP)   +---->|              |
          +------------+ (query) +------------+     +--------------+

http://stable.ascii-flow.appspot.com/#Draw4708199759719921107/990568983


> Lennart
> 
> --
> Lennart Poettering, Red Hat
> _______________________________________________
> systemd-devel mailing list
> systemd-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- 
Holger Winkelmann
Managing Director

email: holger.winkelmann at travelping.com
phone: +49-391-819099-223
mobil: +49-171-5594745
http://www.linkedin.com/in/hwinkel