[systemd-devel] xz troubles

[Reindl Harald]

Am 02.07.2014 02:20, schrieb Zbigniew Jędrzejewski-Szmek:
> this is in some sense a continuation of the subject of xz compression
> for coredumps.
> 
> I've been benchmarking systemd-journal-remote + nc upload speed on
> some fake data in which the MESSAGE fields are a few kb, big enough to
> trigger xz compression in the journal. This is probably not a normal
> stream, but not a completely impossible one. The result is of sending
> 708 MB of logs is:
> - without compression, 713 MB of journal files, 23 s,
> - with compression, 361 MB of journal, 19 min 29 s!
> 
> So the 23 seconds are not particularly impressive, I think there are
> some unnecessary reallocs and copies there which can be eliminated
> improved by implementing a sliding window. I tried to minimize the
> memory usage, but it seems to be hurting performance. But the 19.5
> minutes is rather bad, and I think it is attributable to compression
> of a gazillion relatively small blobs.
> 
> This could also explain why journald sometimes performes so badly.
> A different compressor?

because XZ is a very very bad idea for such things
in case of coredumps LZO/LZ4 would be the right compression

XZ gives a great compression rate but it takes *very long*
see also the fedora-devel thread about deltarpm in case of
XZ compressed packages which are rebuilt locally and take
way longer than download the whole RPM

XZ is hence great for compress one time and use many times
like RPM packages, logrotate where it does not matter if
it takes 30 minutes in the middle of the night but should
be avoided for things user tasks are waiting for

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 246 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freedesktop.org/archives/systemd-devel/attachments/20140702/32ea605b/attachment.sig>