[systemd-devel] journal space requirements

[Kay Sievers]

On Thu, Nov 29, 2012 at 1:42 PM, Colin Guthrie <gmane at colin.guthr.ie> wrote:
> 'Twas brillig, and Colin Guthrie at 29/11/12 12:22 did gyre and gimble:
>> 'Twas brillig, and Zbigniew Jędrzejewski-Szmek at 29/11/12 11:48 did
>> gyre and gimble:
>>> On Thu, Nov 29, 2012 at 10:37:26AM +0000, Colin Guthrie wrote:
>>>> Hi,
>>>>
>>>> So a couple complaints/queries are beginning trickling in regarding
>>>> journal space requirements.
>>>>
>>>> A user was complaining that rotated journals were taking up too much
>>>> room and they could be compressed down etc. I did explain that a rotated
>>>> journal is really any different to the current journal other than it can
>>>> be sealed, but I do fear he has some kind of point regarding long term
>>>> storage.
>>>>
>>>> Should there be some kind of journal archiver system that will run xz -9
>>>> on older journals? Is this something that's being planned or is it left
>>>> as an exercise for the reader to implement such long term
>>>> storage/archiving systems?
>>>
>>> systemd-journal-remote can be used (when merged) to rewrite journal
>>> files, so the tools are mostly there.  Nevertheless, the data in
>>> .journal files should already be compressed with XZ (if enabled), so I
>>> doubt that that would gain much. Do you have some statistics about
>>> the gains from the additional external XZ compression?
>>
>> Hmmm, I wonder if I've missed something obvious in my build then.
>>
>> I just took a cleanly rotated journal of 18M in size and it compressed
>> down to 4M... might be time to look at the build requires :D
>
> OK, so I think I probably was missing an XZ build dep, but even on a
> local build (where I have the devel libs for it already installed), it
> still seems to make a big difference...
>
> -rw-r----- 1 root adm  4.7M Nov 29 12:36 system.journal
> -rw-r--r-- 1 root root 132K Nov 29 12:37 system.journal.xz
>
>
> Am I missing something obvious here. Is "XZ...yes" in the configure
> output enough or do I have to do something special to enable it in the
> journal too?
>
> Or is it just that the uncompressed indexes take up that much space
> generally or that on-the-fly compression just isn't as efficient as post
> rotation compression?

The journal compresses only the data of keys, and only when they are
larger than a certain threshold. It's not surprising that, the
metadata of the journal files can be compressed further to a certain
amount.

But we cannot read the files and do not want to run de-compression on
it while reading the files. Compression could happen transparently in
the filesystem if it supports it, like btrfs. Or archived files would
get moved out of the area where we read stuff can be manually
compressed.

I don't think we want to aim for another level of
rotation/compression, it's actually one of the features that the
databases can be efficiently mapped by the readers. I assume, the cost
of the used space usually beats the effort of a half-hearted built-in
archival that makes the database very expensive and inefficient to
use.

Kay