[systemd-devel] journald on embedded systems

[Chris Morgan]

On Sat, Mar 14, 2015 at 11:07 AM, Zbigniew Jędrzejewski-Szmek
<zbyszek at in.waw.pl> wrote:
> On Sat, Mar 14, 2015 at 10:59:42AM -0400, Chris Morgan wrote:
>> On Fri, Mar 13, 2015 at 11:03 PM, Zbigniew Jędrzejewski-Szmek
>> <zbyszek at in.waw.pl> wrote:
>> > On Fri, Mar 13, 2015 at 06:35:38PM -0400, Chris Morgan wrote:
>> >> Hello.
>> >>
>> >> I posted this, http://lists.freedesktop.org/archives/systemd-devel/2013-July/011926.html,
>> >> some time ago about tiered logging for embedded systems.
>> >>
>> >> The goal is to guarantee that the flash memory will last the duration
>> >> of the product by carefully controlling who writes to it.
>> >>
>> >> I'm back looking at the issue and wanted to re-open the discussion.
>> >>
>> >> One approach that came up would be to set "Storage=volatile", a limit
>> >> of say "10MB" for the journal size, and then write a separate program
>> >> that would filter out the journal entries and write them to a file on
>> >> a physical disk.
>> > You should be able to do something like this right now with journal-remote:
>> > journalctl --directory /run/log/<bootid> -o export | \
>> >    systemd-journal-remote -o /var/log/<bootid>/ -
>> >
>>
>> Thats pretty cool. I didn't realize systemd-journal-remote existed.
>> Using journalctl would mean we have the ability to add filters to the
>> output stream.
> Yes.
>
>> > This is a bit too hacky to put into production, and it would be better
>> > to have a single binary which does this. But all the parts are there:
>> >
>> > - following the journal and filtering,
>> > - opening of specific directory as input,
>> > - saving of state (i.e. the last cursor written),
>> > - writing to a directory and rotating files the same as journald does.
>> >
>> > So adding a new binary (or extending one of the existing ones)
>> > supporting your use case would be a matter of hooking stuff together.
>> >
>> >> The filtering portion is required as we are using the
>> >> journal to persist some important information that we'd like to log.
>> >> We'd also like to preserve high priority messages but don't mind if we
>> >> lose low priority ones across reboots.
>> >>
>> >> An upside of the external program is that we can filter on both high
>> >> priority messages as well as those with specific "FIELD=value"
>> >> entries. Downside is a custom format file and can't use journalctl to
>> >> search through it, no log rotation, no size limits etc.
>> >>
>> >> At the time there was some thought of putting this into journald
>> >> itself. I'm wondering how that would fit given the desire to use more
>> >> complicated matching to decide which entries were put into the
>> >> persisted journal.
>> >
>> > Adding filtering and splitting functionality to journald is another
>> > story of course. It probably would work better: more efficent, and
>> > journal entries would not be duplicated at all.
>> >
>>
>> The piping example you gave does seem to be a bit more heavyweight
>> than a process that was using the sd_journal_xxx() calls to interact
>> with the journal.
>>
>> It doesn't look like systemd-journal-remote supports size limits or
>> rotation, the man page doesn't have any options for that.
> It does. They are not configurable, so it just uses the defaults.
> I guess that this should be added.
>

Ahh. Let me take a look at adding those as a way to figure out how to
test systemd in a container and get started developing.


>> Would it be much less efficient to make yet another program that used
>> sd_journal_xxx() calls, and whatever functionality
>> systemd-journal-remote used to write to a new journal vs. trying to
>> extend journald to do this internally? I'd rather not try to force
>> things into journald proper that don't belong there if the separate
>> application approach is nearly as performant and much cleaner.
> One thing which implementing this in journald would give over implementing
> is in something separate: journald could be configured to split messages
> between /run, or /var. journalctl has the ability to transparently merge
> events from multiple directories, so this split would be invisible to
> users.
>
> I don't see how this could be done so nicely with a separate tool.
>

Yeah. Journalctl could blend the two together. I guess that implies
that the entries put into the persisted journal would not make it into
the non-persisted journal, eg. there wouldn't be duplicate entries.


>> Or possibly enhance systemd-journal-remote to also support following
>> the journal in addition to its current behavior of taking input from
>> stdin?
> Yeah, that seems like a good idea.
>
>> >> If it would fit inside of journald I'd be willing to implement it but
>> >> we would need to come up with a way to configure the filtering, where
>> >> the files are persisted etc. The filtering is a new requirement since
>> >> the last time this was discussed.
>> > I think that supporting a set of similar filters to journalctl would
>> > be a good start. Options which limit the number of messages or
>> > filter based on time would not make sense for journald, but most
>> > other probably would.
>> >
>>
>> Not following what you are referring to when you say "supporting a set
>> of similar filters to journalctl would be a good start".
> _FIELD=value, --priority, --unit, --user-unit, --dmesg, --identifier, --user,
> --system.
>
> Zbyszek

Ahh. Thought it supported a bit of that already.

Chris