[systemd-devel] Empty journal files consume space

[Andrei Borzenkov]

On Thu, Feb 1, 2024 at 3:25 PM Steve Traylen <steve.traylen at cern.ch> wrote:
>
> Hi,
>
> I'm trying to understand why I am only retaining just a couple of days
> of logs when I would like to have more.
>
> The system journalctl head of the logs is only  today:
> Feb 01 10:47:14 nodeX.example.ch systemd-journald[722]: Data hash table
> of /var/log/journal/c33ef6d0ada04ec4abc79c567a7d94b0/system.journal has
> a fill level at 75.0 (174765 of 233016 items, 58720256 file size, 335
> bytes per hash table item), suggesting rotation.
> Feb 01 10:47:14 nodeX.example.ch systemd-journald[722]:
> /var/log/journal/c33ef6d0ada04ec4abc79c567a7d94b0/system.journal:
> Journal header limits reached or header out-of-date, rotating.
>
>
> # journalctl --disk-usage
> Archived and active journals take up 8.1G in the file system.
>
> Reality is  system journal is tiny:
>
> # du -sh system.journal
> 17M     system.journal
>
> However we do have many
>
> # ls -l user-*journal | wc -l
> 1044
>
> and indeed
>
> # du -sh /var/log/journal/c33ef6d0ada04ec4abc79c567a7d94b0
> 8.2G    /var/log/journal/c33ef6d0ada04ec4abc79c567a7d94b0
>
> The vast majority of these user journals are empty and offline
>
> # file user-*journal | awk '{print $4, $5}' | sort | uniq -c
>      940 empty, offline
>      102 offline
>      2 online
>
>
> These user journals are all 8.0M is size
>
> So I think I have two questions:
>
> 1) Why am I loosing old logs sooner than I would like - what limit is "
> fill level at 75.0 (174765 of 233016 items"

You did not provide any evidence that logs are lost. Archived
(offline) logs are processed and searched by journalctl so the oldest
available log is the oldest archive file, not the current online file.

The limit is the fill grade of the hash table in the individual log
file. It is hard coded and unrelated to the limits configured in the
journald.conf. It may affect how long logs are kept if you configured
retention by the number of log files.

> 2) Is there a safe mechanism to delete those empty offline user journals?
>

Just delete them.

> Thanks.
>
> Steve.
>
> Version and configuration:
>
> systemd-252-18.el9 - RHEL9 with a configuration of:
>
> [Journal]
> Storage = persistent
> SplitMode = uid
> SystemMaxUse = 3G
> SystemKeepFree = 10G
> MaxRetentionSec = 1year
>
> # df -h /
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/vda1        80G   65G   16G  81% /
>
>