[systemd-devel] Losing old journal messages after "Journal header limits reached"

[Vadim Zeitlin]

 Hello,

 I'm using systemd 252.19 with default options under Debian Bookworm and am
running into some strange behaviour: even though journal should be able to
consume up to 4GB of space (our of ~500GB available in /var), it only
stores about 800MB of messages and all messages more than a few old are
simply lost, e.g. right now the first message in my journal dates from
2024-01-26T09:11:15 even though

	# journalctl --disk-usage      
	Archived and active journals take up 827.0M in the file system.

and restarting systemd-journald.service says

	System Journal (/var/log/journal/xxx) is 826.9M, max 4.0G, 3.1G free.

 I'd like to understand why does this happen and, equally if not more
importantly, how to prevent it from happening and store more logs. The only
relevant thing I could find were multiples occurrences of messages like this:

	Data hash table of /var/log/journal/xxx/system.journal has a fill level at 75.0 (174763 of 233016 items, 125829120 file size, 719 bytes per hash table item), suggesting rotation.
	/var/log/journal/xxx/system.journal: Journal header limits reached or header out-of-date, rotating.

 I don't see such messages on the other machines, so I suspect they're
related to the problem, but I don't really understand how exactly nor how
to avoid them (other than just logging fewer messages, I guess?). In
particular, even if there are just too many messages and the file needs to
be rotated, why do the older files get deleted instead of being preserved
as long as 4GB of space is consumed?

 What can I do to store more logs? I don't think increasing "Max"
parameters would do me any good, as the existing limits are far from being
reached anyhow, but what else can I try? FWIW I ran "journal --verify"
without finding any problems.

 Thanks in advance for any ideas!
VZ