[systemd-devel] logging in RAM and journald configuration issue

[Dave Howorth]

This message was caught up in what looks like a freedesktop server
crash so I'm reposting it slightly edited after other responses.
Hopefully it will now reach the list and thread somewhat sensibly.

> Lennart Poettering wrote on 20/03/2019 13:37:
> > On Mi, 20.03.19 09:57, Dave Howorth (systemd at howorth.org.uk) wrote:
> >   
> >>> On Di, 19.03.19 20:45, Dave Howorth (systemd at howorth.org.uk)
> >>> wrote: 
> >>>> In the case of a machine that uses an SD card as its primary
> >>>> backing store, it is desirable to reduce the number of write
> >>>> operations to the card in order to prolong its life. journald is
> >>>> quite well-behaved in this regard since it offers the choice of a
> >>>> temporary or permanent journal and limits the frequency of its
> >>>> writes, except in emergencies.
> >>>>
> >>>> However, its permanent journal is written under /var/log/journal
> >>>> and there is no way to configure the path as far as I am aware.
> >>>> I think this is a problem.
> >>>>
> >>>> The reason is that on this type of machine people sometimes map
> >>>>  /var/log to RAM using tmpfs and then perhaps persist the logs
> >>>> using a program like log2ram. When this is done, journald's
> >>>> emergency writing capability is lost and crash analysis becomes
> >>>> more difficult.
> >>>>
> >>>> Of course it is possible instead to redirect the log files of
> >>>> programs individually to temporary memory using systemd-tmpfiles
> >>>> wherever needed. But this involves reconfiguring each and every
> >>>> program that uses /var/log both initially and whenever new
> >>>> programs are installed. This is tedious not only in quantity but
> >>>> because each program has a different detailed format of
> >>>> configuration file.
> >>>>
> >>>> So making /var/log into a tmpfs is a more attractive option. But
> >>>> ideally the journal would be placed somewhere else in persistent
> >>>> storage so its contents are available after a crash. This does
> >>>> not seem to be possible through lack of a config option.
> >>>>
> >>>> Is my analysis correct? Are there any other ways to resolve this
> >>>> difficulty? Otherwise, is it possible to consider a log location
> >>>> config option for journald?  
> >>>
> >>> Right now, when persistent mode is enabled journald will store
> >>> its log data in /var/log. When it is disabled it will store
> >>> things in /run/log instead.
> >>>
> >>> It has been requested that we add a hybrid mode that makes
> >>> journald log to both locations at the same time, but filter by
> >>> log priority so that log msgs higher than some priority go to one
> >>> location and the ones below it go to the other. A patch like that
> >>> would probably be relatively straight-forward and short. Would be
> >>> happy to review/merge a patch for that.
> >>>
> >>> I think if that's implemented the log location should really stay
> >>> unmodified: /var should be persistant and /run not, and there
> >>> would be no need to remount any of those paths in a different
> >>> way.  
> >>
> >> Many thanks for the quick reply. I'm not clear how that resolves
> >> the situation I explained, since it neither provides an alternative
> >> persistent log location nor provides an alternative means of
> >> arranging the logs of other programs. It doesn't seem to address my
> >> issue at all?  
> > 
> > I don't understand the problem then?
> > 
> > The journal only collects logs on stdout/stderr, syslog and its
> > native protocol. Nothing else. It's not a tool that can collect
> > arbitrary per-app logs that don't go via these transport hence. And
> > it really shouldn't be.
> > 
> > Also: /var is generally understood to be persistent, and /run to be
> > volatile. If you want to redefine that you are welcome to, but of
> > course you have to patch through all kinds of software then.  
>
> As I understand it, you want /var/log to be tmpfs but /var/log/journal
> to be persistent (as journald's write behaviour is considerate enough
> for you not to worry about backing store fatigue etc)?

Yes, a very good summary.

> Just as a less complex suggestion, you're presumably bootstrapping
> /var/log to be on tmpfs at some point in early boot (i.e. before
> journald is started or at least before it's being told to start
> writing to /var/log/journal). If this is the case, then why don't you
> just amend that bootstrapping to bind mount /var/log/journal to
> persistent storage rather than letting it be just a subdir on the
> tmpfs? That way you can get your mem-only syslog (or native logging)
> setup as you do now and get persistent journal logs with only very
> minor changes (this could all be done with appropriate systemd units
> and deps AFAICT).

I'm not an expert, or even very familiar, with most of this stuff so I
can't say for sure but yes I think it should be possible to do
something like that.

> I can't remember off the top of my head what makes journald start
> writing to /var/log/journal, so you may have to change that slightly
> (e.g. it might notice is as soon as you mkdir it in the tmpfs but
> before you do the bind mount), but I'm sure this would be simple
> enough.

At present AFAICT the log2ram.service runs
Before=systemd-journald.service and various other services, so I think
that aspect is covered.

[edit]: further to Lennart's later comment. Thanks for that tip. I'll
look to make that change to the dependency.

Many thanks to both for what sounds like a good solution :)