[systemd-devel] /var/log/journal full, journald is not removing journal files

[Chris Murphy]

systemd-238-8.git0e0aa59.fc28.x86_64

I'm really confused by what I'm seeing.

Jul 10 09:13:40 f28h.local systemd-journald[493]: System journal
(/var/log/journal/bbe68372db9f4c589a1f67f008e70864) is 1.2G, max 1.3G,
90.0M free.
[chris at f28h ~]$ du -sh /var/log/journal/bbe68372db9f4c589a1f67f008e70864/
1.6G    /var/log/journal/bbe68372db9f4c589a1f67f008e70864/
[chris at f28h ~]$

[chris at f28h ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p7  1.9G  1.9G   73M  97% /var/log

journald.conf is Fedora default except
SystemMaxUse=2G


1. Somehow systemd-journald is deciding to max at 1.3G, instead of the
specified 2G, which is fine. But I don't know how it arrives at this.
2. Clearly max 1.3G is being busted, the contents of var/log/journal
are greater than the max. non-journal files total ~83M and have not
grown at all in the last week (and multiple reboots).
3. If I change SystemMaxUse=1300M there is no change. No attempt by
journald to clean up /var/log/journal, and no errors, it uses /run/log
instead and never switches to persistent logging.
4. My reading of man journald.conf is that that SystemKeepFree=
defaults to 15% of the file system space, so even with SystemMaxUse=2G
journald should have deleted journal files before getting to 100%
full.
5. If I boot with systemd.log_level=debug, there are no journald
entries that help understand why there's no transition from volatile
to persistent storage, i.e. hey var/log/journal is full, and also that
I can't delete files because $reasons, or whatever.

Extra info: this is a 2G f2fs file system mounted at /var/log. Seems
to be working well except for this little problem but I don't see it
being the cause. journald isn't even attempting to delete its own
journal files to free up space.


Anyway the main thing that has me confused is the max 1.3G statement,
which has been the same since the file system was 5% full upon
creation, and then journals increased all the way to enospc without
any of them being deleted.

-- 
Chris Murphy