[systemd-devel] Linux Journal API/client lib

[Rainer Gerhards]

On Fri, Dec 2, 2011 at 7:47 PM, Kay Sievers <kay.sievers at vrfy.org> wrote:
>>> Well, if syslogd, or any other consumer, is interested in the
>>> metadata, it should not rely in /dev/log. /dev/log will probably stay
>>> what it is which is mostly plain old syslog with a header and a
>>> timestamp and the human readable string. All stuff that wants the
>>> metadata should use the proper API and get the records from there. The
>>> '/dev/log proxy' is just to ensure full syslogd compatibility, not to
>>> provide any new data which do not really fit into the plaintext file
>>> format.
>>
>> Why are you making it deliberately hard to interoperate?
>
> Making what hard? Interoperate? What do you mean? Syslog can very
> easily be a consumer of the journal data, if it's interested in it.

I mean that you make it harder for the syslogd to read the journal
than necessary. Granted, I started this discussion with the assumption
that I needed to write some input plugin that reads the journal and
passes that on (just like we do with the Windows Event Log). But then
I was deligthed to hear that you provide kind of a bridge, where the
system log socket (as seen by syslogd) receives data both from
syslog() as well as your new API. HOWEVER, that log socket seems to be
a a second-class citizen, in that it does not receive all the
information, only the human-readable text. So the most interesting
part (the structured data) is not passed. So in order to obtain the
full information, I still need to read the journal via an additional
plugin, what strictly would not be necessary. This is why I conclude
you are making it hard(er than necessary) to interoperate. I suggest
that you review this decision. But, of course, it's yours.

>> If you
>> already supply the event you gather via the log socket, why not
>> provide the complete set of information (probably as an opt-in
>> feature)? That way the syslogd would not need a specific input
>> capability (input module in rsyslog terms) to capture the log data.
>> Things could just remain as is.
>
> You mean the native log messages only or all messages, including the
> ones journald retrieves from /dev/log

I understood from your posting, that you provide everything that is
logged, not matter what API is used, but only the non-structured data
part.

>> What you call the "plaintext file
>> format" issue is being solved anyhow in the meantime. Look at RFC5424
>> or CEE. So how to encode the fields is actually not a question at all.
>> If you provide proper formatting, the syslogd would not even need to
>> be aware of what's going on... (at least for CEE, rsyslog also has a
>> parser for structured data in legacy syslog, but that's non-standard).
>
> The journal is not meant to replace syslog, if syslog is needed, nor
> to act as a proxy to enrich /dev/log messages. We need to change as
> little as possible, and that includes keeping the same format on the
> output (the fd handed to syslog) as we get at the input side
> (/dev/log).

That sound much like "we got a better interface, but we don't like to
share it" ;)

> If one day glibc changes or there is another generally useful new
> syslog client API, we can do alll that, but as of now, I'm convinced
> that the journal should just make it easy to get all the information
> out, but it should not try to add stuff to established interfaces, and
> it should not natively speak any syslog enhanced format. That stays as
> the job for syslog.

The key point is that you *do not need a new syslog client API*, as
for example CEE was specially modelled to live within the legacy
syslog space. For RFC5424 it would be useful to have a new API, but we
(rsyslog) already got a parser to support RFC5424 STRUCTURED-DATA
within legacy messages. I agree, though, that is it somewhat clumpsy
to do this on top of the syslog() API. But this is exactly where I
thought your effort intended to help. If you known the name-value
pairs, why not format the syslog message in one of these standard
formats?

Don't get me wrong: I agree that some syslog implementation may have
problems with the somewhat longer messages. However, on Linux we have
syslog-ng and rsyslog in most cases. Both handle this very well
(syslog-ng just recently added a JSON parser to support CEE). I think
it would be useful to pass a standards-format syslog message with all
the data you have, provided the system *is configured* to do so. You
may turn this off by default (a conservative choice), but I don't see
a reason to *prevent* passing this information that is readly
available.

Rainer