[Spice-devel] Thoughts about improving streaming video

[John A. Sullivan III]

Hello, all.  Andrea Celestino was kind enough to email me off list about
streaming video performance as reducing SPICE bandwidth consumption for
streaming video is his project as a Computer Engineering student.  He
agreed that I could repost our conversation to invite input from others
on the list.  The (top posted - sorry) thread is below - John


Hi, Andrea.  No problem with your English.  It is certainly better than
my very rusty Italian!

I know little about video other than as a network engineer and I am not
a programmer but I do know what we need.  From that ignorant
perspective, I see several possibilities.

Video is inherently brutal when it comes to network bandwidth and
processing.  There are also different usage scenarios.  For example, it
makes sense to stream when the video needs to be played only once and
started immediately.  However, if it is going to be viewed multiple
times, the Citrix approach of downloading and playing locally makes more
sense.  I've mentioned on the list how we can do this with technology
already in the X2Go project.  It is probably the least desirable option
and the least interesting for your project.  Of course, we have to
handle all this complexity in a way which does not confuse the end user
who just wants to click on their file or web site and have it work!

For streaming the video between the SPICE server and client, I think we
also have a couple of options.  I don't think there is any magic that
can be done.  It is a matter of using the most appropriate codec for the
transmission.  The challenge, as already mentioned on the list, is
encoding on the fly.  The clients need to decode no matter what but, in
non-SPICE usage, the file is pre-encoded.  So, in our selection of
codecs, speed of encoding versus decoding becomes critical.  We must
also not only choose one which uses an open source license but one which
is not patent encumbered.  I believe that rules out X264.  The most
likely candidate is probably VP8 with Theora as an alternative.

Once a codec (or a selection of codecs) has been chosen, the next
challenge is how to switch.  One option is to simply make it a setting
in the SPICE parameters for qemu.  The two advantages are that it is the
simplest and it allows the system administrator (who hopefully knows his
system well) to make the balance between CPU utilization and bandwidth.
The disadvantage is that it does not adapt to the users environment and
may require a reboot to change.

The second option is to select the codec dynamically.  Not only would we
select whether to use a lossy or lossless codec based upon the pixel
change rate, but we would then choose the codec based upon the bandwidth
to the client balanced against the bitrate of the video.  There is
already precedent for this logic in that SPICE already chooses whether
to use lossy or lossless codecs based upon detected bandwidth.  From a
previous post by Marian Krcmarik:

'Once client connects to a guest, "spice server" determines client's
bandwidth (look at code for details, I believe It could be more
appropriate and guys are working on it). "jpeg-wan-compression" and
"zlib-glz-wan-compression" are enabled in case the client's bandwidth is
<10Mbps, otherwise It's disabled. When jpeg compression and zlib over
glz compression are enabled then photo-like bitmaps are compressed by
lossy jpeg compression and textual/artificial bitmaps are compressed by
lossy zlib on top of GLZ.'

SPICE would compare the bandwidth to the bandwidth requirements of the
video and, if the video bandwidth requirements are some percentage less
than the available bandwidth (to leave some of other traffic - I do not
know what that number should be - 80%?), it uses MJPEG for better
quality and lower CPU utilization.  If not, it uses VP8 or whatever we
choose.  This could be multiple choice depending on bitrate/bandwidth if
that makes sense and not just MJPEG vs VP8.

This would be the most interesting approach although we should sacrifice
our programming interest to the best solution for the end users.  It
does have the advantage of not needing to be tweaked by the admin and
adapting to whether the user is on the end of a 3G cell connection or a
100 Mbps WAN link.

As I think about it, we already have precedent for making it the best of
all worlds by using an auto or fixed setting like SPICE does for other
parameters - auto activates the automated routine above and a fixed
value (MJPEG or VP8) specifically locks it to always use the specified
codec.  This is probably a good idea in general but would be
particularly helpful in the early stages while we tweak what that magic
percentage is for allowing other traffic, i.e., does the higher
utilization codec kick in at 80%, 60%, 95%.

Would you mind if I posted this response to the list to see if others
have helpful input? Thanks - John

On Fri, 2011-06-24 at 11:18 +0200, Andrea Celestino wrote:
> Hi,
> 
> first of all I am studying the source coude of spice, because I need
> to understand how spice detects video streaming, how it sends stream
> data to client with socket and how it compress data with mjpeg. 
> I have not any idea at the moment, I need to do some tests. 
> I am very interested.
> Do you have an idea?
> 
> 
> Thanks.
> 
> 
> p.s. sorry for my english :)
> 
> 
> 
> 
> 
> 2011/6/23 John A. Sullivan III <jsullivan at opensourcedevel.com>
>         Certo! Hope I remembered that correctly.  I would be extremely
>         interested. What did you have in mind? Thanks - John
>         
>         On Thu, 2011-06-23 at 10:29 +0200, Andrea Celestino wrote:
>         > Hi,
>         > I have read your message on Spice mailing lists and I am
>         interesting
>         > in streaming video performance with spice. I am an italian
>         student in
>         > computer engineering, during this period I am working on
>         reducing
>         > bandwidth usage with streaming video in Spice. I see that
>         you are
>         > interested in the same problem. I think that we can
>         cooperate. What do
>         > you think about it?
>         
>         
>         
>