[systemd-devel] Unbounded journal~ file accumulation

[Lennart Poettering]

On Wed, 26.11.14 21:59, Moyer, Keith (Keith.Moyer at netapp.com) wrote:

> Running v215-6 via Debian jessie
> 
> In my situation, it is common to have unclean reboots.  On the
> subsequent boot, a system at ....journal~ file is created.  The problem
> is that these journal~ files do not trigger any cleanup with respect
> to SystemMaxUse.  If this occurs repeatedly, disk usage can go _way_
> above SystemMaxUse.

See Zbigniew's comments on more recent fixes to this.

> Sending a SIGUSR2 to the systemd-journald process to force a
> rotation cleans them up, but without that (and assuming the active
> journal file doesn't get full and need to be rotated) there doesn't
> seem to be any limit on how much disk space the journal will use
> (I've had it go 10x SystemMaxUse - until the entire filesystem was
> full - according to du and journalctl --disk-usage).
> 
> I could add a service to my boot that issues the SIGUSR2, but the
> unnecessary _rotation_ would result in a journal file that only had
> the relatively small number of messages that cover part of a single
> bootup sequence (while still using 4M of disk space*).  My
> SystemMaxuse is pretty small (low tens of MiB), so that's a
> significant chunk wasted every boot.
> 
> Is there any way I can trigger the _cleanup_ (vacuum?) without
> triggering the rotation of the active log?  Is the behavior I'm
> seeing a bug that should be filed (should it already be doing that
> cleanup without me having to trigger it)?

There's now "journalctl --vacuum-size=23M" that removes journal files
until the disk usage is under the specified limit. It removes both
the clean and unclean journal files.

> * With my low SystemMaxUse, 4M is pretty large.  I see this minimum
> is hard-coded as JOURNAL_FILE_SIZE_MIN in the code.  Would I be 
> inviting problems if I lowered this value to something like 512K?  
> It was 64K until commit 253f59dff9.  Note: this query is secondary
> to the journal~ accumulation issue.

Well, very small journal files are not efficient. Neither in size
(because we compress identical fields into one instance if they keep
appearing), nor in access time (since access is optimized for few
small rather than many small files, and complexity grows O(n) with
number of files).

Lennart

-- 
Lennart Poettering, Red Hat