[systemd-devel] Linux Journal API/client lib

[Kay Sievers]

On Fri, Dec 2, 2011 at 21:21, Rainer Gerhards <rgerhards at gmail.com> wrote:
> On Fri, Dec 2, 2011 at 7:47 PM, Kay Sievers <kay.sievers at vrfy.org> wrote:
>>>> Well, if syslogd, or any other consumer, is interested in the
>>>> metadata, it should not rely in /dev/log. /dev/log will probably stay
>>>> what it is which is mostly plain old syslog with a header and a
>>>> timestamp and the human readable string. All stuff that wants the
>>>> metadata should use the proper API and get the records from there. The
>>>> '/dev/log proxy' is just to ensure full syslogd compatibility, not to
>>>> provide any new data which do not really fit into the plaintext file
>>>> format.
>>>
>>> Why are you making it deliberately hard to interoperate?
>>
>> Making what hard? Interoperate? What do you mean? Syslog can very
>> easily be a consumer of the journal data, if it's interested in it.
>
> I mean that you make it harder for the syslogd to read the journal
> than necessary. Granted, I started this discussion with the assumption
> that I needed to write some input plugin that reads the journal and
> passes that on (just like we do with the Windows Event Log). But then
> I was deligthed to hear that you provide kind of a bridge, where the
> system log socket (as seen by syslogd) receives data both from
> syslog() as well as your new API. HOWEVER, that log socket seems to be
> a a second-class citizen, in that it does not receive all the
> information, only the human-readable text. So the most interesting
> part (the structured data) is not passed. So in order to obtain the
> full information, I still need to read the journal via an additional
> plugin, what strictly would not be necessary. This is why I conclude
> you are making it hard(er than necessary) to interoperate. I suggest
> that you review this decision. But, of course, it's yours.

Well, that's not making it hard to access information, it's trying
hard not to break established interfaces. If syslog *could* be
extended in a reasonable way, we wouldn't need a journal interface at
all. Adding any information might break tools, so we can only offer
the extended stuff in a new interface.

>>> If you
>>> already supply the event you gather via the log socket, why not
>>> provide the complete set of information (probably as an opt-in
>>> feature)? That way the syslogd would not need a specific input
>>> capability (input module in rsyslog terms) to capture the log data.
>>> Things could just remain as is.
>>
>> You mean the native log messages only or all messages, including the
>> ones journald retrieves from /dev/log
>
> I understood from your posting, that you provide everything that is
> logged, not matter what API is used, but only the non-structured data
> part.

We just forward the exact same information syslog information we have
intercepted, and we will also the syslog compatible part of the native
journal information we received. But we are absolutely not allowed to
change the format of the messages we from /dev/log.

>>> What you call the "plaintext file
>>> format" issue is being solved anyhow in the meantime. Look at RFC5424
>>> or CEE. So how to encode the fields is actually not a question at all.
>>> If you provide proper formatting, the syslogd would not even need to
>>> be aware of what's going on... (at least for CEE, rsyslog also has a
>>> parser for structured data in legacy syslog, but that's non-standard).
>>
>> The journal is not meant to replace syslog, if syslog is needed, nor
>> to act as a proxy to enrich /dev/log messages. We need to change as
>> little as possible, and that includes keeping the same format on the
>> output (the fd handed to syslog) as we get at the input side
>> (/dev/log).
>
> That sound much like "we got a better interface, but we don't like to
> share it" ;)

No, we are just not allowed to break it. As said, if syslog would be
extendable, we wouldn't need all that.

>> If one day glibc changes or there is another generally useful new
>> syslog client API, we can do alll that, but as of now, I'm convinced
>> that the journal should just make it easy to get all the information
>> out, but it should not try to add stuff to established interfaces, and
>> it should not natively speak any syslog enhanced format. That stays as
>> the job for syslog.
>
> The key point is that you *do not need a new syslog client API*, as
> for example CEE was specially modelled to live within the legacy
> syslog space. For RFC5424 it would be useful to have a new API, but we
> (rsyslog) already got a parser to support RFC5424 STRUCTURED-DATA
> within legacy messages. I agree, though, that is it somewhat clumpsy
> to do this on top of the syslog() API. But this is exactly where I
> thought your effort intended to help. If you known the name-value
> pairs, why not format the syslog message in one of these standard
> formats?

Sure, journald can help here, but not by incompatibly pimping up
/dev/log data, but by providing a syslog daemon a data source that has
a well defined format.

> Don't get me wrong: I agree that some syslog implementation may have
> problems with the somewhat longer messages. However, on Linux we have
> syslog-ng and rsyslog in most cases. Both handle this very well
> (syslog-ng just recently added a JSON parser to support CEE). I think
> it would be useful to pass a standards-format syslog message with all
> the data you have, provided the system *is configured* to do so. You
> may turn this off by default (a conservative choice), but I don't see
> a reason to *prevent* passing this information that is readly
> available.

Sure, everything is fully available and easily accessible; just not on
the /dev/log socket, because the additional data does not fit into it
without breaking assumptions, and we need to stay fully compatible
with the current format; journald just can not mangle /dev/log.

Kay