[Spice-devel] [RFC] Spice channels compression

[Frediano Ziglio]

> 
> > 
> > Hi Snir,
> > 
> > Regarding compression and another topic recently evoked, endianness, I
> > wonder
> > if encoding using LEB128 had been discussed?
> > 
> > LEB128 encodes any value below 128 as 1 bytes, any value below ~16000 as 2
> > bytes, and so on. It is size and endianness independent. This means that we
> > can extend a field from uint16 to uint32 without protocol incompatibility.
> > It is much less computationally expensive than full compression. I think it
> > would apply very well to a lot of our meta-data, but would be less
> > effective
> > than regular compression algorithms with image data proper. So it’s
> > addressing another part of the problem.
> > 
> > I have no real feeling yet for how important that other part is in the
> > streams, but my guts tell me “probably not much” (i.e. I expect much more
> > image data than meta-data). Still, it might be worth trying. But it’s
> > probably not just a capability: if we want to encode most fields that way,
> > this means a protocol bump. Not sure it’s worth it.
> > 
> > The code to encode and decode is really short. Here is an example for
> > writing:
> > https://github.com/c3d/XL-programming-language/blob/master/xlr/serializer.cpp#L219.
> > Here is an example for reading:
> > https://github.com/c3d/XL-programming-language/blob/master/xlr/serializer.cpp#L406.
> > 
> > 
> > Christophe
> > 
> 
> Maybe helpful however Snir is working more on payload compression than
> structured data so the gain would be not great (he use few fields).
> However maybe some statistics can be done. Our "mini" header is 16 bit
> type and 32 bit length. For instance you can add some statistics recording
> - number of messages
> - total messages payload
> - total LEB128 encoded header (the not encoded is number of messages * 6).
> If you want also to try compressing (or doing statistics on) fields you
> could add some code to the code generated by python scripts in spice-common.
> 

I instrumented a bit the code to understand how this LEB128 could affect
fields (not the header).
The algorithm remove about 50-60% of field data however this affect the
total bandwidth utilization by about 0.2-0.4% mainly due to the display
channel. Maybe for other channels we could have better gain.

Frediano

> 
> > > On 2 Mar 2017, at 17:53, Snir Sheriber <ssheribe at redhat.com> wrote:
> > > 
> > > This series of patches allows compression of messages over
> > > selected spice channels
> > > 
> > > Few notes:
> > > 
> > > *Currently lz4 stream and regular compression are in use for small and
> > > large
> > > messages accordingly. packets are being sent in common msg type that was
> > > added,
> > > and it utilize previous compressed message structure.
> > > 
> > > *The stream compression & decompression dictionary is based on previous
> > > messages,
> > > there for messages that has compressed\decompressed in stream mode are
> > > being saved
> > > in continuous pre-allocated buffer and a pointer is used for tracking
> > > current
> > > position.
> > > 
> > > *Currently all channels are allowed to be compressed, we might want to
> > > avoid
> > > compression for some channels (for good or just in specific cases)
> > > ***please
> > > notice that usbredir compression was not revert yet so meanwhile i added
> > > 2
> > > small
> > > patches to disable its compression caps.
> > > 
> > > *Multiple clients- basically it should work with more than one client,
> > > although adding
> > > compression/decompression to just part of the clients could theoretically
> > > make it
> > > worse (good for compression performance testing though)
> > > 
> > > -If someone has encountered issues with the combination of compression
> > > and
> > > other spice features please let me know
> > > -Suggestions and comments are welcome, especially for improving the
> > > messages sending  :)
> > > 
> > > Snir.
> > > 
> > > Spice components:
> > > server,client,spice-common,spice-protocol
> > >