[Spice-devel] Thoughts about improving streaming video

[Yaniv Kaul]

Licensing and patent concerns of x264 aside (see 
http://mailman.videolan.org/pipermail/x264-devel/2010-July/007508.html), 
the more I think about it, the more it makes sense to me to have H.264 
offload feature in QXL:
1. Many (Flash, HTML5, others) are moving to H.264 - and take advantage 
of hardware acceleration if such is available, for complete decoding of 
the stream. If QXL advertises itself as being able to HW accel. the 
stream, we need to just pick the stream and move it to the client. On 
the client, we can either download it to the local GPU which will HW 
accelerate its decoding, or use software-based decoding (x264, for example).
2. Then we can revisit x264 and see if it really provides a noticeable 
advantage over mjpeg - clearly having one video solution is cleaner and 
less hassle, and x264 seems to have some nice properties we can use (see 
http://x264dev.multimedia.cx/archives/249 regarding low latency).

Both are not trivial tasks, I'm afraid.
Y.


On 06/25/2011 06:33 PM, John A. Sullivan III wrote:
> Hello, all.  Andrea Celestino was kind enough to email me off list about
> streaming video performance as reducing SPICE bandwidth consumption for
> streaming video is his project as a Computer Engineering student.  He
> agreed that I could repost our conversation to invite input from others
> on the list.  The (top posted - sorry) thread is below - John
>
>
> Hi, Andrea.  No problem with your English.  It is certainly better than
> my very rusty Italian!
>
> I know little about video other than as a network engineer and I am not
> a programmer but I do know what we need.  From that ignorant
> perspective, I see several possibilities.
>
> Video is inherently brutal when it comes to network bandwidth and
> processing.  There are also different usage scenarios.  For example, it
> makes sense to stream when the video needs to be played only once and
> started immediately.  However, if it is going to be viewed multiple
> times, the Citrix approach of downloading and playing locally makes more
> sense.  I've mentioned on the list how we can do this with technology
> already in the X2Go project.  It is probably the least desirable option
> and the least interesting for your project.  Of course, we have to
> handle all this complexity in a way which does not confuse the end user
> who just wants to click on their file or web site and have it work!
>
> For streaming the video between the SPICE server and client, I think we
> also have a couple of options.  I don't think there is any magic that
> can be done.  It is a matter of using the most appropriate codec for the
> transmission.  The challenge, as already mentioned on the list, is
> encoding on the fly.  The clients need to decode no matter what but, in
> non-SPICE usage, the file is pre-encoded.  So, in our selection of
> codecs, speed of encoding versus decoding becomes critical.  We must
> also not only choose one which uses an open source license but one which
> is not patent encumbered.  I believe that rules out X264.  The most
> likely candidate is probably VP8 with Theora as an alternative.
>
> Once a codec (or a selection of codecs) has been chosen, the next
> challenge is how to switch.  One option is to simply make it a setting
> in the SPICE parameters for qemu.  The two advantages are that it is the
> simplest and it allows the system administrator (who hopefully knows his
> system well) to make the balance between CPU utilization and bandwidth.
> The disadvantage is that it does not adapt to the users environment and
> may require a reboot to change.
>
> The second option is to select the codec dynamically.  Not only would we
> select whether to use a lossy or lossless codec based upon the pixel
> change rate, but we would then choose the codec based upon the bandwidth
> to the client balanced against the bitrate of the video.  There is
> already precedent for this logic in that SPICE already chooses whether
> to use lossy or lossless codecs based upon detected bandwidth.  From a
> previous post by Marian Krcmarik:
>
> 'Once client connects to a guest, "spice server" determines client's
> bandwidth (look at code for details, I believe It could be more
> appropriate and guys are working on it). "jpeg-wan-compression" and
> "zlib-glz-wan-compression" are enabled in case the client's bandwidth is
> <10Mbps, otherwise It's disabled. When jpeg compression and zlib over
> glz compression are enabled then photo-like bitmaps are compressed by
> lossy jpeg compression and textual/artificial bitmaps are compressed by
> lossy zlib on top of GLZ.'
>
> SPICE would compare the bandwidth to the bandwidth requirements of the
> video and, if the video bandwidth requirements are some percentage less
> than the available bandwidth (to leave some of other traffic - I do not
> know what that number should be - 80%?), it uses MJPEG for better
> quality and lower CPU utilization.  If not, it uses VP8 or whatever we
> choose.  This could be multiple choice depending on bitrate/bandwidth if
> that makes sense and not just MJPEG vs VP8.
>
> This would be the most interesting approach although we should sacrifice
> our programming interest to the best solution for the end users.  It
> does have the advantage of not needing to be tweaked by the admin and
> adapting to whether the user is on the end of a 3G cell connection or a
> 100 Mbps WAN link.
>
> As I think about it, we already have precedent for making it the best of
> all worlds by using an auto or fixed setting like SPICE does for other
> parameters - auto activates the automated routine above and a fixed
> value (MJPEG or VP8) specifically locks it to always use the specified
> codec.  This is probably a good idea in general but would be
> particularly helpful in the early stages while we tweak what that magic
> percentage is for allowing other traffic, i.e., does the higher
> utilization codec kick in at 80%, 60%, 95%.
>
> Would you mind if I posted this response to the list to see if others
> have helpful input? Thanks - John
>
> On Fri, 2011-06-24 at 11:18 +0200, Andrea Celestino wrote:
>> Hi,
>>
>> first of all I am studying the source coude of spice, because I need
>> to understand how spice detects video streaming, how it sends stream
>> data to client with socket and how it compress data with mjpeg.
>> I have not any idea at the moment, I need to do some tests.
>> I am very interested.
>> Do you have an idea?
>>
>>
>> Thanks.
>>
>>
>> p.s. sorry for my english :)
>>
>>
>>
>>
>>
>> 2011/6/23 John A. Sullivan III<jsullivan at opensourcedevel.com>
>>          Certo! Hope I remembered that correctly.  I would be extremely
>>          interested. What did you have in mind? Thanks - John
>>
>>          On Thu, 2011-06-23 at 10:29 +0200, Andrea Celestino wrote:
>>          >  Hi,
>>          >  I have read your message on Spice mailing lists and I am
>>          interesting
>>          >  in streaming video performance with spice. I am an italian
>>          student in
>>          >  computer engineering, during this period I am working on
>>          reducing
>>          >  bandwidth usage with streaming video in Spice. I see that
>>          you are
>>          >  interested in the same problem. I think that we can
>>          cooperate. What do
>>          >  you think about it?
>>
>>
>>
>>
>
>
> _______________________________________________
> Spice-devel mailing list
> Spice-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/spice-devel