[systemd-devel] Filtering / routing messages to different journal namespaces

[Hannu Lounento]

Hi,

My question would be: is using journal namespaces like described below a 
reasonable approach or abusing the namespace concept? If reasonable, 
would it be possible to extend the `sd_journal` API -- more specifically 
`sd_journal_send` or alike as we'd need to add structured data to the 
entries -- to support writing to a selected namespace?

We have an embedded system where we would like to treat a few categories 
of log messages differently: some should be persisted for later 
retrieval, some sent to an external system (e.g. over the syslog 
protocol) immediately, and the rest / most messages should be stored in 
memory to minimize flash wear.
Furthermore, some messages (from either proprietary or especially 3rd 
party components) should be decorated with additional metadata in 
addition to one of the aforementioned persistence actions. A single 
process may produce log messages that fall into different categories.

One option that we're considering is running journal namespaces with 
different storage configurations for the different categories of 
persistence (but still have all applications log to the default 
namespace), and a separate filtering component (either proprietary or 
some 3rd party log / filtering daemon possibly with custom plugin(s)) 
that would basically follow the default namespace journal, decorate 
matching messages and forward them as necessary to other journal 
namespace(s) for local persistence or other sinks for other storage 
actions like sending to a remote system.

This design, however, would require the filtering component to be able 
to write to the desired journal namespace each message is targeted to, 
which doesn't seem to be possible with the current `sd_journal` API [1] 
(please let us know if this is incorrect). I assume the usual case is to 
route all logging from a given unit directly to a single namespace 
according to LogNamespace= [2] configured for the unit.

A fallback option would of course be to implement a custom storage for 
the persisted messages if nothing else fits the use case, but we'd 
preferably take advantage of systemd-journald's features including e.g. 
structured log entries, rotation & other storage options, and the 
`sd_journal` API for reading and filtering messages.

[1] https://www.freedesktop.org/software/systemd/man/sd-journal.html
[2] 
https://www.freedesktop.org/software/systemd/man/systemd.exec.html#LogNamespace=

Thanks,
Hannu Lounento
hannu.lounento at vaisala.com