[Spice-devel] [RFC] Spice channels compression

[Snir Sheriber]

Hi,

Thanks all for your comments


On 03/03/2017 12:49 PM, Frediano Ziglio wrote:
>> Hey,
>>
>> Do you have any measurements of how much bandwidth is saved on each
>> channel type on basic usage of linux/windows guests? Ideally, with
>> numbers about additional CPU usage and added latency?
>>
>> Christophe
>>
> Yes, without numbers it's a bit hard understand if worst or not.

The core channels i aimed to were usbredir,webdav and main, i haven't
came across yet with significant results in other channels.
The avg latency is about 0.2-0.3 ms per packet when large packet are being
transferred (tested on usbredir), btw i didn't try anything with windows 
guest yet,
might be interesting

Numbers are pretty dependent on what the user is doing, for example:
( Number are the compression rate = uncompressed_size/compressed_size)

1. usbredir: connecting usb-4GB storage device with ext4 (no files transfer)
server -> client:
2.6
client -> server:
8.2 (4MB)

2.webdav : copy compressed JPEG client-> server
server -> client:
1.27
client -> server:
1.7 (copy jpg)

3. main : It start with impressive compression but afterwards it mostly 
depends on user behavior , for example copy\paste of text should be 
compressed massively in main channel
server -> client:
150.1?! (but just for the first msgs)
client -> server:
2.9

4. cursor channel behaves similarly- large compression during boot 
(using client
mouse after boot) and that's it

in other words i would describe compression rate for these channels as:
Usbredir- mostly device dependent, could achieved high compression in 
some common (maybe even the most common) use cases (better than current 
compression)
webdav\main - it depends what the user is doing\transfer, could achieve
high compression ratio as well
>> On Thu, Mar 02, 2017 at 06:53:23PM +0200, Snir Sheriber wrote:
>>> This series of patches allows compression of messages over
>>> selected spice channels
>>>
> Which channels did you try?
> Do you have statistics for various channels?
> I suppose for instance that for sound and display (which are already compressed)
> is not much worth doing while for instance cursor and usb would gain
> more (cursor shapes are not currently compressed).
>
>>> Few notes:
>>>
>>> *Currently lz4 stream and regular compression are in use for small and
>>> large
>>> messages accordingly. packets are being sent in common msg type that was
> Why did you do such distinction for small and large?
> Have you tried to just use stream all the time?

Stream compression limitations.. the very large one are pretty rare..
>>> added,
>>> and it utilize previous compressed message structure.
>>>
>>> *The stream compression & decompression dictionary is based on previous
>>> messages,
>>> there for messages that has compressed\decompressed in stream mode are
>>> being saved
>>> in continuous pre-allocated buffer and a pointer is used for tracking
>>> current
>>> position.
>>>
> Wouldn't be worth trying to use framing functions?
> LZ4 framing functions already handle the buffer for you. They also
> support flush which would be useful to support large messages (looks
> like reading the code we are limited to 64K).
> Would be also possibly worth trying to move the channel compression
> to lower level (RedsStream) to compress everything and possibly
> multiple messages together.

I tried it before on packages that was captured from spice channels
iirc performance wasn't bad but stream compression was better.. (cpu\
latency & compression rate)
I guess it is possible to do something like that (similar idea as the 
tls ?),
i would say that if it will be implemented in lower level the framing 
lz4 may
be a wiser choice.
btw this idea has came up before but i was slightly preferred the 
current one that
utilize the protocol messages and the already exists compression 
structure. i guess it
would take some time but i could try it that way also and make measurements
afterwards..

>>> *Currently all channels are allowed to be compressed, we might want to
>>> avoid
>>> compression for some channels (for good or just in specific cases)
>>> ***please
>>> notice that usbredir compression was not revert yet so meanwhile i added 2
>>> small
>>> patches to disable its compression caps.
>>>
> Why you had to disable usb ?

The idea is to revert the usbredir-compression if these patches would be
upstream eventually, so it's just to avoid double usb compression for now..
>>> *Multiple clients- basically it should work with more than one client,
>>> although adding
>>> compression/decompression to just part of the clients could theoretically
>>> make it
>>> worse (good for compression performance testing though)
>>>
> It's not clear this point

ahh, never mind it wont work on the impotent channels anyway
>>> -If someone has encountered issues with the combination of compression and
>>> other spice features please let me know
>>> -Suggestions and comments are welcome, especially for improving the
>>> messages sending  :)
>>>
>>> Snir.
>>>
>>> Spice components:
>>> server,client,spice-common,spice-protocol
>>>
> Frediano
Snir.