[systemd-devel] Empty journal files consume space

[Steve Traylen]

On 01/02/2024 13:45, Andrei Borzenkov wrote:

> On Thu, Feb 1, 2024 at 3:25 PM Steve Traylen <steve.traylen at cern.ch> wrote:
>> Hi,
>>
>> I'm trying to understand why I am only retaining just a couple of days
>> of logs when I would like to have more.
>>
>> The system journalctl head of the logs is only  today:
>> Feb 01 10:47:14 nodeX.example.ch systemd-journald[722]: Data hash table
>> of /var/log/journal/c33ef6d0ada04ec4abc79c567a7d94b0/system.journal has
>> a fill level at 75.0 (174765 of 233016 items, 58720256 file size, 335
>> bytes per hash table item), suggesting rotation.
>> Feb 01 10:47:14 nodeX.example.ch systemd-journald[722]:
>> /var/log/journal/c33ef6d0ada04ec4abc79c567a7d94b0/system.journal:
>> Journal header limits reached or header out-of-date, rotating.
>>
>>
>> # journalctl --disk-usage
>> Archived and active journals take up 8.1G in the file system.
>>
>> Reality is  system journal is tiny:
>>
>> # du -sh system.journal
>> 17M     system.journal
>>
>> However we do have many
>>
>> # ls -l user-*journal | wc -l
>> 1044
>>
>> and indeed
>>
>> # du -sh /var/log/journal/c33ef6d0ada04ec4abc79c567a7d94b0
>> 8.2G    /var/log/journal/c33ef6d0ada04ec4abc79c567a7d94b0
>>
>> The vast majority of these user journals are empty and offline
>>
>> # file user-*journal | awk '{print $4, $5}' | sort | uniq -c
>>       940 empty, offline
>>       102 offline
>>       2 online
>>
>>
>> These user journals are all 8.0M is size
>>
>> So I think I have two questions:
>>
>> 1) Why am I loosing old logs sooner than I would like - what limit is "
>> fill level at 75.0 (174765 of 233016 items"
> You did not provide any evidence that logs are lost. Archived
> (offline) logs are processed and searched by journalctl so the oldest
> available log is the oldest archive file, not the current online file.
>
> The limit is the fill grade of the hash table in the individual log
> file. It is hard coded and unrelated to the limits configured in the
> journald.conf. It may affect how long logs are kept if you configured
> retention by the number of log files.
Thanks for reply.

There are no archive files I believe:

# ls /var/log/journal/514fed82c54d4a89b9f7f8f33eca1c8e/*system*
/var/log/journal/514fed82c54d4a89b9f7f8f33eca1c8e/system.journal

The archive files would be alongside the live file I believe.

Just tried an explicit " journalctl --rotate" which logs:

Feb 01 14:36:33 nodeX.example.ch systemd-journald[658]: System Journal 
(/var/log/journal/514fed82c54d4a89b9f7f8f33eca1c8e) is 8.0G, max 3.0G, 
0B free.
Feb 01 14:36:40 nodeX.example.ch systemd-journald[658]: Received client 
request to rotate journal, rotating.
Feb 01 14:36:40nodeX.example.ch systemd-journald[658]: Deleted empty 
archived journal 
/var/log/journal/514fed82c54d4a89b9f7f8f33eca1c8e/user-1234 at 537a18390e124dd6b4cf41a69ef5780d-0000000000000000-0000000000000000.journal 
(3.5M).
Feb 01 14:36:40 lxplus978.cern.ch systemd-journald[658]: Deleted empty 
archived journal 
/var/log/journal/514fed82c54d4a89b9f7f8f33eca1c8e/user-1235 at d7d23966c1454001a714ee5aef039c60-0000000000000000-0000000000000000.journal 
(3.5M).

So now maybe I understand at rotation I am over the configured max of 
3GB so perhaps no archive is generated. Looking at another node with 
fewer number of users having ever logged in I have the archive of
of the system log and a longer history. Those 940 "empty, offline"  user 
journals consume the space providing no particular value.

No other indication that rotation may not have worked.


>> 2) Is there a safe mechanism to delete those empty offline user journals?
>>
> Just delete them.
>
>> Thanks.
>>
>> Steve.
>>
>> Version and configuration:
>>
>> systemd-252-18.el9 - RHEL9 with a configuration of:
>>
>> [Journal]
>> Storage = persistent
>> SplitMode = uid
>> SystemMaxUse = 3G
>> SystemKeepFree = 10G
>> MaxRetentionSec = 1year
>>
>> # df -h /
>> Filesystem      Size  Used Avail Use% Mounted on
>> /dev/vda1        80G   65G   16G  81% /
>>
>>