[systemd-devel] xz troubles

[Lennart Poettering]

On Wed, 02.07.14 02:20, Zbigniew Jędrzejewski-Szmek (zbyszek at in.waw.pl) wrote:

> Hi,
> 
> this is in some sense a continuation of the subject of xz compression
> for coredumps.
> 
> I've been benchmarking systemd-journal-remote + nc upload speed on
> some fake data in which the MESSAGE fields are a few kb, big enough to
> trigger xz compression in the journal. This is probably not a normal
> stream, but not a completely impossible one. The result is of sending
> 708 MB of logs is:
> - without compression, 713 MB of journal files, 23 s,
> - with compression, 361 MB of journal, 19 min 29 s!
> 
> So the 23 seconds are not particularly impressive, I think there are
> some unnecessary reallocs and copies there which can be eliminated
> improved by implementing a sliding window. I tried to minimize the
> memory usage, but it seems to be hurting performance. But the 19.5
> minutes is rather bad, and I think it is attributable to compression
> of a gazillion relatively small blobs.
> 
> This could also explain why journald sometimes performes so badly.
> 
> A different compressor?

Hmm, I think we can be smarter with the XZ hookup first. Currently, we
don't attempt to reuse the XZ context (has been on the TODO list for
long). I have the suspicion that could have a major effect on
performance.

Lennart

-- 
Lennart Poettering, Red Hat