<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> <p class="MsoNormal">Thanks! Persisting the /etc/machine-id file on firmware updates will resolve this issue. <o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal"><b>From:</b> Mantas Mikulėnas <grawity@gmail.com> <br> <b>Sent:</b> Tuesday, March 1, 2022 9:53 AM<br> <b>To:</b> Zaluzec, Eric <Eric.Zaluzec@vertiv.com><br> <b>Cc:</b> systemd-devel@lists.freedesktop.org<br> <b>Subject:</b> Re: [systemd-devel] Journald: Re-use /var/log/journal hash entry after system upgrade<o:p></o:p></p> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> <div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt"> <p class="MsoNormal" style="line-height:12.0pt;background:#FFEB9C"><span style="font-size:10.0pt;color:#9C6500">CAUTION:</span><span style="font-size:10.0pt;color:black"> This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.<o:p></o:p></span></p> </div> <p class="MsoNormal"><o:p> </o:p></p> <div> <div> <div> <p class="MsoNormal">On Tue, Mar 1, 2022 at 4:39 PM <a href="mailto:Eric.Zaluzec@vertiv.com"> Eric.Zaluzec@vertiv.com</a> <<a href="mailto:Eric.Zaluzec@vertiv.com">Eric.Zaluzec@vertiv.com</a>> wrote:<o:p></o:p></p> </div> <div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"> <div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">#### [ System Environment ] ####<br> On an embedded x86-64 Linux system, I’m running systemd v241. I have Systemd-Journald logging set to persistent with SystemMaxUse and RuntimeMaxUse both set to 512MB. <o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The Linux system mount loops /var/log on start-up from a var-log.ext4 file. The /var/log mount is given a fixed size of the disk (976MB). Systemd creates a journal entry directory given a hash name in /var/log/journal/<hash1>.<o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">#### [ System Information ] ####<o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Linux 4.19.0-6-2-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux<o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br> #### [ Problem ] ####<o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">When the system firmware is upgraded, Systemd creates a new journal entry directory with a different hash name and no longer recognizes the previous hash directory. <o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">The old logs from the previous journal entry can no longer be managed. The old logs are never rotated and cannot be manually rotated using the journaldctl cli. The disk usage calculator by Journald does not account for the previous journal entry meaning if there are two previous entries in /var/log that consume 900MB of space; then the new journal entry only has 76MB of space to work with. Eventually, disk space will be full. Journald cannot automatically flush, vacuum, clean, rotate previous logs because it does not recognize the previous journal entries.<o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">There must be a systemd journald check that occurs where it determines these other entries are not for it to manage.<o:p></o:p></p> </div> </div> </blockquote> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">This is not a 'hash' – it's the <a href="https://www.freedesktop.org/software/systemd/man/machine-id.html"> machine ID</a>, which is directly read from <span style="font-family:"Courier New""> /etc/machine-id</span>. Normally the machine ID is randomly generated during <i>first </i>boot (it's just a random UUID), and it is supposed to be persistent afterwards.<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">It sounds like your <span style="font-family:"Courier New""> /etc</span> doesn't have that file and is on a read-only rootfs, so systemd generates a new machine-id in tmpfs every boot. The device probably has *some* persistent storage though, so try to find a way to make /etc/machine-id persist as well (see the <a href="https://www.freedesktop.org/software/systemd/man/machine-id.html">machine-id(5)</a> manual page for a few possibilities).<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">The intent of using machine-id subdirectories is to allow containers' journal directories to be symlinked on the host, or remote system journals to be collected on a single system – there's actually the <span style="font-family:"Courier New"">journalctl -m</span> option to make it look at "foreign" journals, so e.g. <span style="font-family:"Courier New"">journalctl -m -f</span> could be used to watch logs of all containers.<o:p></o:p></p> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <div> <p class="MsoNormal">(The machine ID is also used as the base for systemd-networkd's DHCPv4 Client ID, DHCPv6 DUID, and IPv6 address generation. But other than that, though, I can't think of any its uses that would be visible externally – aside from desktop-specific things like pulseaudio – so in some cases it might be "good enough" to pre-define a fixed ID in the template image as a last resort.)<o:p></o:p></p> </div> </div> <div> <p class="MsoNormal"><o:p> </o:p></p> </div> <p class="MsoNormal">-- <o:p></o:p></p> <div> <div> <p class="MsoNormal">Mantas Mikulėnas<o:p></o:p></p> </div> </div> </div> </div> </div> CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. </body> </html>