[Spice-devel] lz4 and streaming compression

[Christophe Fergeau]

Hey,

On Wed, Jan 27, 2016 at 09:18:14AM -0500, Frediano Ziglio wrote:
> Hi,
>   after the analysis of image performance and looking at the way data are sent
> I was thinking about compression at stream level (that is socket instead of image
> level).

Makes sense, though we probably do not want to always compress (eg
already-compressed audio streams in the record/playback channels).

> This would have the advantage to compress entire traffic (even headers)
> and better usage of dictionaries. Currently lz4 is not taking advantage of image
> compressing data just as sequences of bytes. The dictionary is not reused and
> reset at every images decreasing compression ratio.

Iirc quic is designed to take advantage of 'patterns' specific to image
data, I don't know if compressing a stream instead of separate images
will make a difference (though maybe quic isn't that good for
compression in the first place, so this could be moot).

> 
> There is however one problem in the current implementation that make difficult
> to implement this. Compression algorithms compress data block wise. That is when
> you keep adding data to a compressed stream data are compressed when a block is
> full. This way client won't receive data unless block is filled. To use compression
> algorithms for streaming (like -C option in ssh which uses gzip algorithm) usually
> a flush operation is implemented. This operation compress data even if block is not
> full. Obviously doing too much flush (like in our case can happen if we compress
> for every message we are sending) reduces compression ratio.
> 
> Beside compression I think I'm going to implement flush to use cork feature on
> network layer. I already did some test time ago and this reduces network bandwidth
> usage. Combined with my patches to remove push needs would possibly lower
> bandwidth usage even more.
> 
> What people think about?

On some channels, we need to be careful about latency (input, audio), so
need some kind of balance between not flushing too often, and not
flushing often enough.
Definitely an interesting experiment if not too complicated/too long.

Christophe
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.freedesktop.org/archives/spice-devel/attachments/20160127/1c89ffe7/attachment.sig>