[Spice-devel] lz4 and streaming compression

[Frediano Ziglio]

> 
> Hey,
> 
> On Wed, Jan 27, 2016 at 09:18:14AM -0500, Frediano Ziglio wrote:
> > Hi,
> >   after the analysis of image performance and looking at the way data are
> >   sent
> > I was thinking about compression at stream level (that is socket instead of
> > image
> > level).
> 
> Makes sense, though we probably do not want to always compress (eg
> already-compressed audio streams in the record/playback channels).
> 

Yes, one problem is video streaming which is in the same channel of display
with images (which can be compressed).

> > This would have the advantage to compress entire traffic (even headers)
> > and better usage of dictionaries. Currently lz4 is not taking advantage of
> > image
> > compressing data just as sequences of bytes. The dictionary is not reused
> > and
> > reset at every images decreasing compression ratio.
> 
> Iirc quic is designed to take advantage of 'patterns' specific to image
> data, I don't know if compressing a stream instead of separate images
> will make a difference (though maybe quic isn't that good for
> compression in the first place, so this could be moot).
> 

One problem is that you restart for every image.
Actually I was thinking you have a glz but not a glz4. I think would be
possible to keep a small dictionary for lz4 not creating the stream for
every image but just at the beginning.

> > 
> > There is however one problem in the current implementation that make
> > difficult
> > to implement this. Compression algorithms compress data block wise. That is
> > when
> > you keep adding data to a compressed stream data are compressed when a
> > block is
> > full. This way client won't receive data unless block is filled. To use
> > compression
> > algorithms for streaming (like -C option in ssh which uses gzip algorithm)
> > usually
> > a flush operation is implemented. This operation compress data even if
> > block is not
> > full. Obviously doing too much flush (like in our case can happen if we
> > compress
> > for every message we are sending) reduces compression ratio.
> > 
> > Beside compression I think I'm going to implement flush to use cork feature
> > on
> > network layer. I already did some test time ago and this reduces network
> > bandwidth
> > usage. Combined with my patches to remove push needs would possibly lower
> > bandwidth usage even more.
> > 
> > What people think about?
> 
> On some channels, we need to be careful about latency (input, audio), so
> need some kind of balance between not flushing too often, and not
> flushing often enough.
> Definitely an interesting experiment if not too complicated/too long.
> 
> Christophe
> 

The client can choose the compression algorithm to use and server can always
send data uncompressed. About compression you could having a flush for every
piece of data you send. Compression ratio will be lower but you can reduce
bandwidth. Considering that a not so good connection usually have higher latency
and lower bandwidth you could even have some gain.
Anyway... without experiments and results it's hard to say.
Perhaps to simplify one way to just enable all traffic and get some data could
be easier instead of implementing all solution (obviously results will be worst
than the optimal solution).

My main idea was for display channel, disabling image compression but compress
at network (application) level.
Everything started realizing that lz4 was just a bytes compression
(quic/jpeg/glz/lz all use image compression).

Frediano