[systemd-devel] [PATCH 2/3] coredump: use lz4frame api to compress coredumps

[Zbigniew Jędrzejewski-Szmek]

On Mon, Dec 08, 2014 at 03:36:15PM +0100, Lennart Poettering wrote:
> On Sun, 07.12.14 19:32, Zbigniew Jędrzejewski-Szmek (zbyszek at in.waw.pl) wrote:
> 
> > This converts the stream compression to use the new lz4frame api,
> > compatible with lz4cat. Previous code used custom headers, so the
> > compressed file was not compatible with lz4 command line tools.
> > I considered this the last blocker to using lz4 by default.
> > 
> > Speed seems to be reasonable, although a bit (a few percent) slower
> > than the lz4 binary, even though compression is the same. I don't
> > consider this important. It could be caused by the overhead of library
> > calls, but is probably caused by slightly different buffer sizes or
> > such. The code in this patch uses mmap, since since this allows the
> > buffer to be reused while not making the code more complicated at all.
> > In my testing, this version is noticably faster (~20%) than a naive
> > single-buffered version. mmap can cause the program to be killed with
> > SIGBUS, if the underlying file is truncated or a disk error occurs. We
> > only use this from within coredump and coredumpctl, so I don't
> > consider this an issue.
> > 
> > Old decompression code is retained and is used if the new code fails
> > indicating a format error. I have various coredumps in the old format
> > and would be unhappy if they suddenly stopped working. We can remove
> > the legacy code in a few versions.
> > 
> > The way that blobs are compressed in the journal is not affected.
> > 
> > Note that lz4 125 has not been released yet, so this patch should
> > not be applied currently.
> 
> Looks all goot to me. What I wonder though is whether we should maybe
> one day compress blobs on the client-side and pass the compressed to
> the journal, instead of compress them inside journald, maybe as a
> later protocol extension?
We could do that. Nevertheless, for lz4 the compression speed is at
30% of memory speed on my machine, so the gains are not that significant.
But going the other way, and allowing the client to request compressed
data could be more useful. Sometimes they are only interested in partial
decompression, so this could provide significant memory and cpu savings.

Zbyszek