[systemd-devel] Journald: Re-use /var/log/journal hash entry after system upgrade

[Eric.Zaluzec at vertiv.com]

Thanks! Persisting the /etc/machine-id file on firmware updates will resolve this issue.

From: Mantas Mikulėnas <grawity at gmail.com>
Sent: Tuesday, March 1, 2022 9:53 AM
To: Zaluzec, Eric <Eric.Zaluzec at vertiv.com>
Cc: systemd-devel at lists.freedesktop.org
Subject: Re: [systemd-devel] Journald: Re-use /var/log/journal hash entry after system upgrade

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

On Tue, Mar 1, 2022 at 4:39 PM Eric.Zaluzec at vertiv.com<mailto:Eric.Zaluzec at vertiv.com> <Eric.Zaluzec at vertiv.com<mailto:Eric.Zaluzec at vertiv.com>> wrote:
#### [ System Environment ] ####
On an embedded x86-64 Linux system, I’m running systemd v241. I have Systemd-Journald logging set to persistent with SystemMaxUse and RuntimeMaxUse both set to 512MB.
The Linux system mount loops /var/log on start-up from a var-log.ext4 file. The /var/log mount is given a fixed size of the disk (976MB). Systemd creates a journal entry directory given a hash name in /var/log/journal/<hash1>.

#### [ System Information ] ####
Linux 4.19.0-6-2-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux

#### [ Problem ] ####
When the system firmware is upgraded, Systemd creates a new journal entry directory with a different hash name and no longer recognizes the previous hash directory.
The old logs from the previous journal entry can no longer be managed. The old logs are never rotated and cannot be manually rotated using the journaldctl cli. The disk usage calculator by Journald does not account for the previous journal entry meaning if there are two previous entries in /var/log that consume 900MB of space; then the new journal entry only has 76MB of space to work with. Eventually, disk space will be full. Journald cannot automatically flush, vacuum, clean, rotate previous logs because it does not recognize the previous journal entries.
There must be a systemd journald check that occurs where it determines these other entries are not for it to manage.

This is not a 'hash' – it's the machine ID<https://www.freedesktop.org/software/systemd/man/machine-id.html>, which is directly read from /etc/machine-id. Normally the machine ID is randomly generated during first boot (it's just a random UUID), and it is supposed to be persistent afterwards.

It sounds like your /etc doesn't have that file and is on a read-only rootfs, so systemd generates a new machine-id in tmpfs every boot. The device probably has *some* persistent storage though, so try to find a way to make /etc/machine-id persist as well (see the machine-id(5)<https://www.freedesktop.org/software/systemd/man/machine-id.html> manual page for a few possibilities).

The intent of using machine-id subdirectories is to allow containers' journal directories to be symlinked on the host, or remote system journals to be collected on a single system – there's actually the journalctl -m option to make it look at "foreign" journals, so e.g. journalctl -m -f could be used to watch logs of all containers.

(The machine ID is also used as the base for systemd-networkd's DHCPv4 Client ID, DHCPv6 DUID, and IPv6 address generation. But other than that, though, I can't think of any its uses that would be visible externally – aside from desktop-specific things like pulseaudio – so in some cases it might be "good enough" to pre-define a fixed ID in the template image as a last resort.)

--
Mantas Mikulėnas
CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/systemd-devel/attachments/20220301/4821b125/attachment-0001.htm>