[Spice-devel] Thoughts about improving streaming video

[John A. Sullivan III]

On Mon, 2011-06-27 at 08:32 +0300, Yaniv Kaul wrote:
> Licensing and patent concerns of x264 aside (see 
> http://mailman.videolan.org/pipermail/x264-devel/2010-July/007508.html), 
> the more I think about it, the more it makes sense to me to have H.264 
> offload feature in QXL:

This sounds great (gaming on SPICE!) but that's a pretty big aside.  How
would we handle the patent issues? - John

> 1. Many (Flash, HTML5, others) are moving to H.264 - and take advantage 
> of hardware acceleration if such is available, for complete decoding of 
> the stream. If QXL advertises itself as being able to HW accel. the 
> stream, we need to just pick the stream and move it to the client. On 
> the client, we can either download it to the local GPU which will HW 
> accelerate its decoding, or use software-based decoding (x264, for example).
> 2. Then we can revisit x264 and see if it really provides a noticeable 
> advantage over mjpeg - clearly having one video solution is cleaner and 
> less hassle, and x264 seems to have some nice properties we can use (see 
> http://x264dev.multimedia.cx/archives/249 regarding low latency).
> 
> Both are not trivial tasks, I'm afraid.
> Y.
> 
> 
> On 06/25/2011 06:33 PM, John A. Sullivan III wrote:
> > Hello, all.  Andrea Celestino was kind enough to email me off list about
> > streaming video performance as reducing SPICE bandwidth consumption for
> > streaming video is his project as a Computer Engineering student.  He
> > agreed that I could repost our conversation to invite input from others
> > on the list.  The (top posted - sorry) thread is below - John
> >
> >
> > Hi, Andrea.  No problem with your English.  It is certainly better than
> > my very rusty Italian!
> >
> > I know little about video other than as a network engineer and I am not
> > a programmer but I do know what we need.  From that ignorant
> > perspective, I see several possibilities.
> >
> > Video is inherently brutal when it comes to network bandwidth and
> > processing.  There are also different usage scenarios.  For example, it
> > makes sense to stream when the video needs to be played only once and
> > started immediately.  However, if it is going to be viewed multiple
> > times, the Citrix approach of downloading and playing locally makes more
> > sense.  I've mentioned on the list how we can do this with technology
> > already in the X2Go project.  It is probably the least desirable option
> > and the least interesting for your project.  Of course, we have to
> > handle all this complexity in a way which does not confuse the end user
> > who just wants to click on their file or web site and have it work!
> >
> > For streaming the video between the SPICE server and client, I think we
> > also have a couple of options.  I don't think there is any magic that
> > can be done.  It is a matter of using the most appropriate codec for the
> > transmission.  The challenge, as already mentioned on the list, is
> > encoding on the fly.  The clients need to decode no matter what but, in
> > non-SPICE usage, the file is pre-encoded.  So, in our selection of
> > codecs, speed of encoding versus decoding becomes critical.  We must
> > also not only choose one which uses an open source license but one which
> > is not patent encumbered.  I believe that rules out X264.  The most
> > likely candidate is probably VP8 with Theora as an alternative.
> >
> > Once a codec (or a selection of codecs) has been chosen, the next
> > challenge is how to switch.  One option is to simply make it a setting
> > in the SPICE parameters for qemu.  The two advantages are that it is the
> > simplest and it allows the system administrator (who hopefully knows his
> > system well) to make the balance between CPU utilization and bandwidth.
> > The disadvantage is that it does not adapt to the users environment and
> > may require a reboot to change.
> >
> > The second option is to select the codec dynamically.  Not only would we
> > select whether to use a lossy or lossless codec based upon the pixel
> > change rate, but we would then choose the codec based upon the bandwidth
> > to the client balanced against the bitrate of the video.  There is
> > already precedent for this logic in that SPICE already chooses whether
> > to use lossy or lossless codecs based upon detected bandwidth.  From a
> > previous post by Marian Krcmarik:
> >
> > 'Once client connects to a guest, "spice server" determines client's
> > bandwidth (look at code for details, I believe It could be more
> > appropriate and guys are working on it). "jpeg-wan-compression" and
> > "zlib-glz-wan-compression" are enabled in case the client's bandwidth is
> > <10Mbps, otherwise It's disabled. When jpeg compression and zlib over
> > glz compression are enabled then photo-like bitmaps are compressed by
> > lossy jpeg compression and textual/artificial bitmaps are compressed by
> > lossy zlib on top of GLZ.'
> >
> > SPICE would compare the bandwidth to the bandwidth requirements of the
> > video and, if the video bandwidth requirements are some percentage less
> > than the available bandwidth (to leave some of other traffic - I do not
> > know what that number should be - 80%?), it uses MJPEG for better
> > quality and lower CPU utilization.  If not, it uses VP8 or whatever we
> > choose.  This could be multiple choice depending on bitrate/bandwidth if
> > that makes sense and not just MJPEG vs VP8.
> >
> > This would be the most interesting approach although we should sacrifice
> > our programming interest to the best solution for the end users.  It
> > does have the advantage of not needing to be tweaked by the admin and
> > adapting to whether the user is on the end of a 3G cell connection or a
> > 100 Mbps WAN link.
> >
> > As I think about it, we already have precedent for making it the best of
> > all worlds by using an auto or fixed setting like SPICE does for other
> > parameters - auto activates the automated routine above and a fixed
> > value (MJPEG or VP8) specifically locks it to always use the specified
> > codec.  This is probably a good idea in general but would be
> > particularly helpful in the early stages while we tweak what that magic
> > percentage is for allowing other traffic, i.e., does the higher
> > utilization codec kick in at 80%, 60%, 95%.
> >
> > Would you mind if I posted this response to the list to see if others
> > have helpful input? Thanks - John
> >
> > On Fri, 2011-06-24 at 11:18 +0200, Andrea Celestino wrote:
> >> Hi,
> >>
> >> first of all I am studying the source coude of spice, because I need
> >> to understand how spice detects video streaming, how it sends stream
> >> data to client with socket and how it compress data with mjpeg.
> >> I have not any idea at the moment, I need to do some tests.
> >> I am very interested.
> >> Do you have an idea?
> >>
> >>
> >> Thanks.
> >>
> >>
> >> p.s. sorry for my english :)
> >>
> >>
> >>
> >>
> >>
> >> 2011/6/23 John A. Sullivan III<jsullivan at opensourcedevel.com>
> >>          Certo! Hope I remembered that correctly.  I would be extremely
> >>          interested. What did you have in mind? Thanks - John
> >>
> >>          On Thu, 2011-06-23 at 10:29 +0200, Andrea Celestino wrote:
> >>          >  Hi,
> >>          >  I have read your message on Spice mailing lists and I am
> >>          interesting
> >>          >  in streaming video performance with spice. I am an italian
> >>          student in
> >>          >  computer engineering, during this period I am working on
> >>          reducing
> >>          >  bandwidth usage with streaming video in Spice. I see that
> >>          you are
> >>          >  interested in the same problem. I think that we can
> >>          cooperate. What do
> >>          >  you think about it?
> >>
> >>
> >>
> >>
> >
> >
> > _______________________________________________
> > Spice-devel mailing list
> > Spice-devel at lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/spice-devel
>