[systemd-devel] Journald: Re-use /var/log/journal hash entry after system upgrade

[Mantas Mikulėnas]

On Tue, Mar 1, 2022 at 4:39 PM Eric.Zaluzec at vertiv.com <
Eric.Zaluzec at vertiv.com> wrote:

> #### [ System Environment ] ####
> On an embedded x86-64 Linux system, I’m running systemd v241. I have
> Systemd-Journald logging set to persistent with SystemMaxUse and
> RuntimeMaxUse both set to 512MB.
>
> The Linux system mount loops /var/log on start-up from a var-log.ext4
> file. The /var/log mount is given a fixed size of the disk (976MB). Systemd
> creates a journal entry directory given a hash name in
> /var/log/journal/<hash1>.
>
>
>
> #### [ System Information ] ####
>
> Linux 4.19.0-6-2-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64
> GNU/Linux
>
>
> #### [ Problem ] ####
>
> When the system firmware is upgraded, Systemd creates a new journal entry
> directory with a different hash name and no longer recognizes the previous
> hash directory.
>
> The old logs from the previous journal entry can no longer be managed. The
> old logs are never rotated and cannot be manually rotated using the
> journaldctl cli. The disk usage calculator by Journald does not account for
> the previous journal entry meaning if there are two previous entries in
> /var/log that consume 900MB of space; then the new journal entry only has
> 76MB of space to work with. Eventually, disk space will be full. Journald
> cannot automatically flush, vacuum, clean, rotate previous logs because it
> does not recognize the previous journal entries.
>
> There must be a systemd journald check that occurs where it determines
> these other entries are not for it to manage.
>

This is not a 'hash' – it's the machine ID
<https://www.freedesktop.org/software/systemd/man/machine-id.html>, which
is directly read from /etc/machine-id. Normally the machine ID is randomly
generated during *first *boot (it's just a random UUID), and it is supposed
to be persistent afterwards.

It sounds like your /etc doesn't have that file and is on a read-only
rootfs, so systemd generates a new machine-id in tmpfs every boot. The
device probably has *some* persistent storage though, so try to find a way
to make /etc/machine-id persist as well (see the machine-id(5)
<https://www.freedesktop.org/software/systemd/man/machine-id.html> manual
page for a few possibilities).

The intent of using machine-id subdirectories is to allow containers'
journal directories to be symlinked on the host, or remote system journals
to be collected on a single system – there's actually the journalctl -m
option to make it look at "foreign" journals, so e.g. journalctl -m -f
could be used to watch logs of all containers.

(The machine ID is also used as the base for systemd-networkd's DHCPv4
Client ID, DHCPv6 DUID, and IPv6 address generation. But other than that,
though, I can't think of any its uses that would be visible externally –
aside from desktop-specific things like pulseaudio – so in some cases it
might be "good enough" to pre-define a fixed ID in the template image as a
last resort.)

-- 
Mantas Mikulėnas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20220301/eb163414/attachment.htm>