[systemd-devel] journal space requirements

[Colin Guthrie]

'Twas brillig, and Kay Sievers at 29/11/12 13:04 did gyre and gimble:
> On Thu, Nov 29, 2012 at 1:42 PM, Colin Guthrie <gmane at colin.guthr.ie> wrote:
>> 'Twas brillig, and Colin Guthrie at 29/11/12 12:22 did gyre and gimble:
>>> 'Twas brillig, and Zbigniew Jędrzejewski-Szmek at 29/11/12 11:48 did
>>> gyre and gimble:
>>>> On Thu, Nov 29, 2012 at 10:37:26AM +0000, Colin Guthrie wrote:
>>>>> Hi,
>>>>>
>>>>> So a couple complaints/queries are beginning trickling in regarding
>>>>> journal space requirements.
>>>>>
>>>>> A user was complaining that rotated journals were taking up too much
>>>>> room and they could be compressed down etc. I did explain that a rotated
>>>>> journal is really any different to the current journal other than it can
>>>>> be sealed, but I do fear he has some kind of point regarding long term
>>>>> storage.
>>>>>
>>>>> Should there be some kind of journal archiver system that will run xz -9
>>>>> on older journals? Is this something that's being planned or is it left
>>>>> as an exercise for the reader to implement such long term
>>>>> storage/archiving systems?
>>>>
>>>> systemd-journal-remote can be used (when merged) to rewrite journal
>>>> files, so the tools are mostly there.  Nevertheless, the data in
>>>> .journal files should already be compressed with XZ (if enabled), so I
>>>> doubt that that would gain much. Do you have some statistics about
>>>> the gains from the additional external XZ compression?
>>>
>>> Hmmm, I wonder if I've missed something obvious in my build then.
>>>
>>> I just took a cleanly rotated journal of 18M in size and it compressed
>>> down to 4M... might be time to look at the build requires :D
>>
>> OK, so I think I probably was missing an XZ build dep, but even on a
>> local build (where I have the devel libs for it already installed), it
>> still seems to make a big difference...
>>
>> -rw-r----- 1 root adm  4.7M Nov 29 12:36 system.journal
>> -rw-r--r-- 1 root root 132K Nov 29 12:37 system.journal.xz
>>
>>
>> Am I missing something obvious here. Is "XZ...yes" in the configure
>> output enough or do I have to do something special to enable it in the
>> journal too?
>>
>> Or is it just that the uncompressed indexes take up that much space
>> generally or that on-the-fly compression just isn't as efficient as post
>> rotation compression?
> 
> The journal compresses only the data of keys, and only when they are
> larger than a certain threshold. It's not surprising that, the
> metadata of the journal files can be compressed further to a certain
> amount.
> 
> But we cannot read the files and do not want to run de-compression on
> it while reading the files. Compression could happen transparently in
> the filesystem if it supports it, like btrfs. Or archived files would
> get moved out of the area where we read stuff can be manually
> compressed.

Yeah, I wouldn't suggest any kind of on-the-fly decompression here. Just
a useful hook point to move the files for long term archival. e.g.
eventually it would be nice if apache logs etc. could go into the
journal (or a dedicated journal) as it would make searching for times,
servers, IP addresses etc. super nice. But I have to keep such logs for
a long time, so it would be good to have a nice mechanism to sync up
when they were put into an archive. They won't be read often, or
(ideally) at all, but a big folder full of such compressed journals
would be ideal.

> I don't think we want to aim for another level of
> rotation/compression, it's actually one of the features that the
> databases can be efficiently mapped by the readers. I assume, the cost
> of the used space usually beats the effort of a half-hearted built-in
> archival that makes the database very expensive and inefficient to
> use.

What would be good was if the rotation of journals allowed some kind of
simple hook to be called. Perhaps just as simple as it calling a
templated systemd unit. Then if someone wanted to implement some kind of
archival system, they could simply just drop that unit in place and
knock themselves out with whatever madcap scheme they could come up with.

Cheers

Col


-- 

Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
  Tribalogic Limited http://www.tribalogic.net/
Open Source:
  Mageia Contributor http://www.mageia.org/
  PulseAudio Hacker http://www.pulseaudio.org/
  Trac Hacker http://trac.edgewall.org/