[systemd-devel] Empty journal files consume space

[Steve Traylen]

Hi,

I'm trying to understand why I am only retaining just a couple of days 
of logs when I would like to have more.

The system journalctl head of the logs is only  today:
Feb 01 10:47:14 nodeX.example.ch systemd-journald[722]: Data hash table 
of /var/log/journal/c33ef6d0ada04ec4abc79c567a7d94b0/system.journal has 
a fill level at 75.0 (174765 of 233016 items, 58720256 file size, 335 
bytes per hash table item), suggesting rotation.
Feb 01 10:47:14 nodeX.example.ch systemd-journald[722]: 
/var/log/journal/c33ef6d0ada04ec4abc79c567a7d94b0/system.journal: 
Journal header limits reached or header out-of-date, rotating.


# journalctl --disk-usage
Archived and active journals take up 8.1G in the file system.

Reality is  system journal is tiny:

# du -sh system.journal
17M     system.journal

However we do have many

# ls -l user-*journal | wc -l
1044

and indeed

# du -sh /var/log/journal/c33ef6d0ada04ec4abc79c567a7d94b0
8.2G    /var/log/journal/c33ef6d0ada04ec4abc79c567a7d94b0

The vast majority of these user journals are empty and offline

# file user-*journal | awk '{print $4, $5}' | sort | uniq -c
     940 empty, offline
     102 offline
     2 online


These user journals are all 8.0M is size

So I think I have two questions:

1) Why am I loosing old logs sooner than I would like - what limit is " 
fill level at 75.0 (174765 of 233016 items"
2) Is there a safe mechanism to delete those empty offline user journals?

Thanks.

Steve.

Version and configuration:

systemd-252-18.el9 - RHEL9 with a configuration of:

[Journal]
Storage = persistent
SplitMode = uid
SystemMaxUse = 3G
SystemKeepFree = 10G
MaxRetentionSec = 1year

# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        80G   65G   16G  81% /