[systemd-devel] logging in RAM and journald configuration issue

[Colin Guthrie]

Lennart Poettering wrote on 20/03/2019 13:37:
> On Mi, 20.03.19 09:57, Dave Howorth (systemd at howorth.org.uk) wrote:
> 
>>> On Di, 19.03.19 20:45, Dave Howorth (systemd at howorth.org.uk) wrote:
>>>
>>>> In the case of a machine that uses an SD card as its primary backing
>>>> store, it is desirable to reduce the number of write operations to
>>>> the card in order to prolong its life. journald is quite
>>>> well-behaved in this regard since it offers the choice of a
>>>> temporary or permanent journal and limits the frequency of its
>>>> writes, except in emergencies.
>>>>
>>>> However, its permanent journal is written under /var/log/journal and
>>>> there is no way to configure the path as far as I am aware. I think
>>>> this is a problem.
>>>>
>>>> The reason is that on this type of machine people sometimes map
>>>>  /var/log to RAM using tmpfs and then perhaps persist the logs
>>>> using a program like log2ram. When this is done, journald's
>>>> emergency writing capability is lost and crash analysis becomes
>>>> more difficult.
>>>>
>>>> Of course it is possible instead to redirect the log files of
>>>> programs individually to temporary memory using systemd-tmpfiles
>>>> wherever needed. But this involves reconfiguring each and every
>>>> program that uses /var/log both initially and whenever new programs
>>>> are installed. This is tedious not only in quantity but because
>>>> each program has a different detailed format of configuration file.
>>>>
>>>> So making /var/log into a tmpfs is a more attractive option. But
>>>> ideally the journal would be placed somewhere else in persistent
>>>> storage so its contents are available after a crash. This does not
>>>> seem to be possible through lack of a config option.
>>>>
>>>> Is my analysis correct? Are there any other ways to resolve this
>>>> difficulty? Otherwise, is it possible to consider a log location
>>>> config option for journald?
>>>
>>> Right now, when persistent mode is enabled journald will store its log
>>> data in /var/log. When it is disabled it will store things in /run/log
>>> instead.
>>>
>>> It has been requested that we add a hybrid mode that makes journald
>>> log to both locations at the same time, but filter by log priority so
>>> that log msgs higher than some priority go to one location and the
>>> ones below it go to the other. A patch like that would probably be
>>> relatively straight-forward and short. Would be happy to review/merge
>>> a patch for that.
>>>
>>> I think if that's implemented the log location should really stay
>>> unmodified: /var should be persistant and /run not, and there would be
>>> no need to remount any of those paths in a different way.
>>
>> Many thanks for the quick reply. I'm not clear how that resolves the
>> situation I explained, since it neither provides an alternative
>> persistent log location nor provides an alternative means of
>> arranging the logs of other programs. It doesn't seem to address my
>> issue at all?
> 
> I don't understand the problem then?
> 
> The journal only collects logs on stdout/stderr, syslog and its native
> protocol. Nothing else. It's not a tool that can collect arbitrary
> per-app logs that don't go via these transport hence. And it really
> shouldn't be.
> 
> Also: /var is generally understood to be persistent, and /run to be
> volatile. If you want to redefine that you are welcome to, but of
> course you have to patch through all kinds of software then.

As I understand it, you want /var/log to be tmpfs but /var/log/journal
to be persistent (as journald's write behaviour is considerate enough
for you not to worry about backing store fatigue etc)?

Just as a less complex suggestion, you're presumably bootstrapping
/var/log to be on tmpfs at some point in early boot (i.e. before
journald is started or at least before it's being told to start writing
to /var/log/journal). If this is the case, then why don't you just amend
that bootstrapping to bind mount /var/log/journal to persistent storage
rather than letting it be just a subdir on the tmpfs? That way you can
get your mem-only syslog (or native logging) setup as you do now and get
persistent journal logs with only very minor changes (this could all be
done with appropriate systemd units and deps AFAICT).

I can't remember off the top of my head what makes journald start
writing to /var/log/journal, so you may have to change that slightly
(e.g. it might notice is as soon as you mkdir it in the tmpfs but before
you do the bind mount), but I'm sure this would be simple enough.

Just a thought.

Col

-- 

Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
  Tribalogic Limited http://www.tribalogic.net/
Open Source:
  Mageia Contributor http://www.mageia.org/
  PulseAudio Hacker http://www.pulseaudio.org/
  Trac Hacker http://trac.edgewall.org/