[systemd-devel] journal space requirements

[Zbigniew Jędrzejewski-Szmek]

On Thu, Nov 29, 2012 at 12:42:32PM +0000, Colin Guthrie wrote:
> 'Twas brillig, and Colin Guthrie at 29/11/12 12:22 did gyre and gimble:
> > 'Twas brillig, and Zbigniew Jędrzejewski-Szmek at 29/11/12 11:48 did
> > gyre and gimble:
> >> On Thu, Nov 29, 2012 at 10:37:26AM +0000, Colin Guthrie wrote:
> >>> Hi,
> >>>
> >>> So a couple complaints/queries are beginning trickling in regarding
> >>> journal space requirements.
> >>>
> >>> A user was complaining that rotated journals were taking up too much
> >>> room and they could be compressed down etc. I did explain that a rotated
> >>> journal is really any different to the current journal other than it can
> >>> be sealed, but I do fear he has some kind of point regarding long term
> >>> storage.
> >>>
> >>> Should there be some kind of journal archiver system that will run xz -9
> >>> on older journals? Is this something that's being planned or is it left
> >>> as an exercise for the reader to implement such long term
> >>> storage/archiving systems?
> >>
> >> systemd-journal-remote can be used (when merged) to rewrite journal
> >> files, so the tools are mostly there.  Nevertheless, the data in
> >> .journal files should already be compressed with XZ (if enabled), so I
> >> doubt that that would gain much. Do you have some statistics about
> >> the gains from the additional external XZ compression?
> > 
> > Hmmm, I wonder if I've missed something obvious in my build then.
> > 
> > I just took a cleanly rotated journal of 18M in size and it compressed
> > down to 4M... might be time to look at the build requires :D
> 
> OK, so I think I probably was missing an XZ build dep, but even on a
> local build (where I have the devel libs for it already installed), it
> still seems to make a big difference...
> 
> -rw-r----- 1 root adm  4.7M Nov 29 12:36 system.journal
> -rw-r--r-- 1 root root 132K Nov 29 12:37 system.journal.xz
> 
> 
> Am I missing something obvious here. Is "XZ...yes" in the configure
> output enough or do I have to do something special to enable it in the
> journal too?
Should be enough.

> Or is it just that the uncompressed indexes take up that much space
> generally or that on-the-fly compression just isn't as efficient as post
> rotation compression?
Yeah, definitely something is not working.

journalctl --lines=100 -o export | build-remote/systemd-journal-remote --compress --stdin -o /tmp/dir2/file.journal

gives me 644K,
and XZ turns that into 31K.

journalctl --lines=100 -o export | build-remote/systemd-journal-remote --no-compress --stdin -o /tmp/dir3/file.journal

gives me 660K,
and XZ turns that into 29K.

Maybe it's just that this is mostly small text messages and compressing
the individually doesn't help much.

Zbyszek