[systemd-devel] journald on embedded systems

[Chris Morgan]

On Fri, Mar 13, 2015 at 11:03 PM, Zbigniew Jędrzejewski-Szmek
<zbyszek at in.waw.pl> wrote:
> On Fri, Mar 13, 2015 at 06:35:38PM -0400, Chris Morgan wrote:
>> Hello.
>>
>> I posted this, http://lists.freedesktop.org/archives/systemd-devel/2013-July/011926.html,
>> some time ago about tiered logging for embedded systems.
>>
>> The goal is to guarantee that the flash memory will last the duration
>> of the product by carefully controlling who writes to it.
>>
>> I'm back looking at the issue and wanted to re-open the discussion.
>>
>> One approach that came up would be to set "Storage=volatile", a limit
>> of say "10MB" for the journal size, and then write a separate program
>> that would filter out the journal entries and write them to a file on
>> a physical disk.
> You should be able to do something like this right now with journal-remote:
> journalctl --directory /run/log/<bootid> -o export | \
>    systemd-journal-remote -o /var/log/<bootid>/ -
>

Thats pretty cool. I didn't realize systemd-journal-remote existed.
Using journalctl would mean we have the ability to add filters to the
output stream.



> This is a bit too hacky to put into production, and it would be better
> to have a single binary which does this. But all the parts are there:
>
> - following the journal and filtering,
> - opening of specific directory as input,
> - saving of state (i.e. the last cursor written),
> - writing to a directory and rotating files the same as journald does.
>
> So adding a new binary (or extending one of the existing ones)
> supporting your use case would be a matter of hooking stuff together.
>
>> The filtering portion is required as we are using the
>> journal to persist some important information that we'd like to log.
>> We'd also like to preserve high priority messages but don't mind if we
>> lose low priority ones across reboots.
>>
>> An upside of the external program is that we can filter on both high
>> priority messages as well as those with specific "FIELD=value"
>> entries. Downside is a custom format file and can't use journalctl to
>> search through it, no log rotation, no size limits etc.
>>
>> At the time there was some thought of putting this into journald
>> itself. I'm wondering how that would fit given the desire to use more
>> complicated matching to decide which entries were put into the
>> persisted journal.
>
> Adding filtering and splitting functionality to journald is another
> story of course. It probably would work better: more efficent, and
> journal entries would not be duplicated at all.
>

The piping example you gave does seem to be a bit more heavyweight
than a process that was using the sd_journal_xxx() calls to interact
with the journal.

It doesn't look like systemd-journal-remote supports size limits or
rotation, the man page doesn't have any options for that.

Would it be much less efficient to make yet another program that used
sd_journal_xxx() calls, and whatever functionality
systemd-journal-remote used to write to a new journal vs. trying to
extend journald to do this internally? I'd rather not try to force
things into journald proper that don't belong there if the separate
application approach is nearly as performant and much cleaner.

Or possibly enhance systemd-journal-remote to also support following
the journal in addition to its current behavior of taking input from
stdin?


>> If it would fit inside of journald I'd be willing to implement it but
>> we would need to come up with a way to configure the filtering, where
>> the files are persisted etc. The filtering is a new requirement since
>> the last time this was discussed.
> I think that supporting a set of similar filters to journalctl would
> be a good start. Options which limit the number of messages or
> filter based on time would not make sense for journald, but most
> other probably would.
>

Not following what you are referring to when you say "supporting a set
of similar filters to journalctl would be a good start".

Chris