[systemd-devel] live remote logging?

[Lennart Poettering]

1;3401;0cOn Wed, 31.10.12 09:48, Cliff Brake (cliff.brake at gmail.com) wrote:

> Hello,
> 
> We have a system that is composed of 8 embedded ARM CPUs, and one server.
>  The embedded ARM units are booting an initramfs that uses systemd.
> 
> We would like to store the logs from the 8 embedded systems on the server.
>  In the past, we would use the syslog networking logging.
> 
> What type of architecture should we use?  Ideally we'd like to be able to
> use the live remote logging that is discussed in the systemd design
> document.

The journal currently has not nice way to do live logging, but we are
working on that. The closest you can get is via: "systemctl enable
systemd-journal-gatewayd.service" and then "curl
foobar:19531/entries?follow" which will give you live output of the logs
of host "foobar".

> I've come up with several ideas:
> 
> 1) scp/rsync the journal files to the server.
>    * how do we know when a file is ready to be scp'd?

The idea is that it is ready at any time. It will be incomplete, but
that should be completely OK. The reader should handle this just fine,
and read as far as it can. I expect that there will probably a couple of
issues with this right now, since this is little used so far. More
specifically, the file parser might choke on incomplete files even
though the general design of it is actually supposed to read anything
you give it.

>    * is there any way to get reasonable update rate?  It seems running
> rsync every 10s might get expensive.

repeated rsync is an option, but yes, it is expensive.

> 2) use the systemd-journal-gatewayd in v195
>    * is there any way to stream the journal with this, or is this more of a
> query interface?

You can stream with it. If you add "?follow" to the URL you get an HTTP
stream that never ends. With the "Range:" HTTP header you can restart
downloads from where you left off, and via "Accept: application/json"
you can request JSON rather then the classic /var/log/messages format.

Depending on what precisely you intend to do with the data on the
centralized host you could of course use the JSON/HTTP iface to browse
the data ad-hoc...

>    * I assume, we then we loose the journal file format

Our plan is to have a client side tool (probably called
systemd-journal-remote) or so, which you can use to push or pull data
from another host, either sporadically when the user needs to, or
continously for live syncing. In the most common case it would use HTTP
and serilize things into a journal file, that should be pretty much
identicall to the one from the server side. 

> 3) write our own app that monitors the journal and copies over data
>    * how would this operate, would we use mmap + file change notifications,
> or journal C api?

The journal C API can do this already. 

>    * with mmap, would we watch all the files in the journal directory?

Yes, that's the usual approach. Either you use the system journal
directory, or you specify another.

> 4) wait for the live remote logging features
>    * any timeline for when this will be available?

There's not much missing actually. The HTTP serializer we already
have. All that's missing is an HTTP deserializer for journal blobs. It's
probably 200 lines of code only. It's actually high on my TODO list,
since I like items to hack on where you can achieve a lot by doing
relatively little... ;-)

> 5) run syslog along side journal and use the syslog network logging features
>    * then the anti-systemd members of the team say -- why don't we just
> ditch systemd and go back to sysvinit and syslog ...  :-)
>    * then we lose the query benefits of the journal on the server.

This would definitely work, and if you need something yesterday, it's a
good choice.

> Additionally, on the server, it seems like it makes sense to use the
> journal files for the log storage.
>   1) how would log rotation be managed (discard old logs)?

You can control rotation by size or by time.

>   2) can the systemd-journal-gatewayd be used on a independent directory of
> log files?

Currently not, but that should be trivial to add it needed.

Lennart

-- 
Lennart Poettering - Red Hat, Inc.