[systemd-devel] [ANNOUNCE] Journal File Format Documentation

[Ciprian Dorin Craciun]

On Tue, Oct 23, 2012 at 7:33 PM, Lennart Poettering
<lennart at poettering.net> wrote:
> On Tue, 23.10.12 18:48, Ciprian Dorin Craciun (ciprian.craciun at gmail.com) wrote:
>
>> > - We needed something with in-line compression, and where we can add
>> >   stuff like FSS to
>>
>>     Ok. I agree that there are very few libraries that fit here. All I
>> can think of making into here would be BerkeleyDB (but it fails other
>> requirements you've listed below).
>
> I used BDB in one other project and it makes me shudder. The constant
> ABI breakages and disk corruptions where aful. Heck, yum as one user of
> it breaks every second Fedora release with a BDB error where the usual
> recipe is to remove the yum BDB database so that it is regenerated on
> next invocation. I am pretty much through with BDB.

    :) Yes, now I remember... Each time I upgrade BerkeleyDB `isync`
and `bogofilter` gets me in hell...


> So, the idea here is really to just index everything, and make it cheap
> so that if something actually never made sense to be indexed is cheap.
>
> In the journal file format indexing field objects that are only
> referenced once is practically free, as instead of storing an offset to
> the bsearch object we use for indexing we just store the offset of the
> referencing entry in-line.

    I guess those offsets are quite cheap, and the in-line entry for
the "once-only" data are ok. But (from what I understand) every value
you store has to be searched through the file before storing (to see
if it already exists as a value). Thus wouldn't this impact CPU usage?


>>     BTW, a little bit off-topic:
>>     * why didn't you try to implement this journal system as a
>> standalone library, that could have been reused by other systems
>> independently of systemd; (I know this was answered in [2], and that
>> the focus is on systemd, but it seems it took quite a lot of work, and
>> it's a pity it can't be individually reused);
>
> Well, for starters we want to be careful where we guarantee interface
> stability and where not. For example, the C API gives you a very
> high-level view on the journals, where interleaving of multiple files is
> hidden. However, if we'd split this out then we'd have to expose much
> more guts of the implementation, and provide stable API for that, and
> that's something I don't want. WE want the ability to change the
> internals of the implementation around and guarantee stability only at
> the most high level C API that hides it all.
>
> The other thing is simply that the stuff is really integrated with each
> other. The journal sources are small because we reuse a lot of internal
> C APIs of systemd, and the format exposes a lot of things that are
> specific to systemd, for example the vocabulary of well-known fields is
> closely bound to systemd.

    I understand this issue with the focus. Nevertheless your journal
idea sounds nice, and I hope someone will take it and implement it in
a standalone variant. (I hope in a native compilable language...)


>>     * how do you intend to implement something resembly syslog's log
>> centralization?
>
> The network model existing since day one is one where we rely on
> existing file sharing infrastructure to transfer/collect files. I.e. use
> NFS, SMB, FTP, WebDAV, SCP, rsync whatever suits you, and make available
> at one spot, and "journactl -m" will interleave them as necessary.

    By "interleave" you mean only "taking note of new files", not
actually "rewriting the contents".

    About the NFS (and other shared FS's) as storage backend I'm not
that certain... (And the same about the rest of `scp`, `rsync`, etc.)

    Maybe a command / option (maybe backed by a `ssh` channel a-la
`rsync`) to fetch efficiently journal from other machines. (Not as you
describe below, which seems geared towards integration, but strictly
geared towards collection.)


> I am curently working on getting log syncing via both a PUSH and PULL
> model done. This will be based one existing protocols and standards as
> much as we can (SSH or HTTP/HTTPS as transport, and JSON and more as
> payload), and is flexible for others to hook into. For example, I think
> it would be cool if greylog2 and similar software would just pull the
> data out of the journal on its own, simply via HTTP/JSON. We make
> everything available to make this smooth, i.e. we provide clients with
> stable cursors which they can use to restart operation.

    Aha. Kind of answers my previous question.

    Thanks for your time,
    Ciprian.