[systemd-devel] journald on embedded systems

[Zbigniew Jędrzejewski-Szmek]

On Sat, Mar 14, 2015 at 10:59:42AM -0400, Chris Morgan wrote:
> On Fri, Mar 13, 2015 at 11:03 PM, Zbigniew Jędrzejewski-Szmek
> <zbyszek at in.waw.pl> wrote:
> > On Fri, Mar 13, 2015 at 06:35:38PM -0400, Chris Morgan wrote:
> >> Hello.
> >>
> >> I posted this, http://lists.freedesktop.org/archives/systemd-devel/2013-July/011926.html,
> >> some time ago about tiered logging for embedded systems.
> >>
> >> The goal is to guarantee that the flash memory will last the duration
> >> of the product by carefully controlling who writes to it.
> >>
> >> I'm back looking at the issue and wanted to re-open the discussion.
> >>
> >> One approach that came up would be to set "Storage=volatile", a limit
> >> of say "10MB" for the journal size, and then write a separate program
> >> that would filter out the journal entries and write them to a file on
> >> a physical disk.
> > You should be able to do something like this right now with journal-remote:
> > journalctl --directory /run/log/<bootid> -o export | \
> >    systemd-journal-remote -o /var/log/<bootid>/ -
> >
> 
> Thats pretty cool. I didn't realize systemd-journal-remote existed.
> Using journalctl would mean we have the ability to add filters to the
> output stream.
Yes.

> > This is a bit too hacky to put into production, and it would be better
> > to have a single binary which does this. But all the parts are there:
> >
> > - following the journal and filtering,
> > - opening of specific directory as input,
> > - saving of state (i.e. the last cursor written),
> > - writing to a directory and rotating files the same as journald does.
> >
> > So adding a new binary (or extending one of the existing ones)
> > supporting your use case would be a matter of hooking stuff together.
> >
> >> The filtering portion is required as we are using the
> >> journal to persist some important information that we'd like to log.
> >> We'd also like to preserve high priority messages but don't mind if we
> >> lose low priority ones across reboots.
> >>
> >> An upside of the external program is that we can filter on both high
> >> priority messages as well as those with specific "FIELD=value"
> >> entries. Downside is a custom format file and can't use journalctl to
> >> search through it, no log rotation, no size limits etc.
> >>
> >> At the time there was some thought of putting this into journald
> >> itself. I'm wondering how that would fit given the desire to use more
> >> complicated matching to decide which entries were put into the
> >> persisted journal.
> >
> > Adding filtering and splitting functionality to journald is another
> > story of course. It probably would work better: more efficent, and
> > journal entries would not be duplicated at all.
> >
> 
> The piping example you gave does seem to be a bit more heavyweight
> than a process that was using the sd_journal_xxx() calls to interact
> with the journal.
> 
> It doesn't look like systemd-journal-remote supports size limits or
> rotation, the man page doesn't have any options for that.
It does. They are not configurable, so it just uses the defaults.
I guess that this should be added.

> Would it be much less efficient to make yet another program that used
> sd_journal_xxx() calls, and whatever functionality
> systemd-journal-remote used to write to a new journal vs. trying to
> extend journald to do this internally? I'd rather not try to force
> things into journald proper that don't belong there if the separate
> application approach is nearly as performant and much cleaner.
One thing which implementing this in journald would give over implementing
is in something separate: journald could be configured to split messages
between /run, or /var. journalctl has the ability to transparently merge
events from multiple directories, so this split would be invisible to
users.

I don't see how this could be done so nicely with a separate tool.

> Or possibly enhance systemd-journal-remote to also support following
> the journal in addition to its current behavior of taking input from
> stdin?
Yeah, that seems like a good idea.

> >> If it would fit inside of journald I'd be willing to implement it but
> >> we would need to come up with a way to configure the filtering, where
> >> the files are persisted etc. The filtering is a new requirement since
> >> the last time this was discussed.
> > I think that supporting a set of similar filters to journalctl would
> > be a good start. Options which limit the number of messages or
> > filter based on time would not make sense for journald, but most
> > other probably would.
> >
> 
> Not following what you are referring to when you say "supporting a set
> of similar filters to journalctl would be a good start".
_FIELD=value, --priority, --unit, --user-unit, --dmesg, --identifier, --user,
--system.

Zbyszek