[Spice-devel] Thoughts about improving streaming video
Yaniv Kaul
ykaul at redhat.com
Mon Jun 27 02:05:23 PDT 2011
On 06/27/2011 11:50 AM, Christophe Fergeau wrote:
> Hey,
>
> Fwiw,
> http://blogs.adobe.com/penguinswf/2010/01/solving_different_problems.html
This is a bit outdated - as they have realized some of the video could
be fully accelerated.
Above: "The Flash Player has to do a little bit more. In addition to
decoding the data, it has to convert YUV data to the RGB colorspace
<http://en.wikipedia.org/wiki/RGB_color_space> and combine the image
with other Flash elements."
And then in
http://www.adobe.com/devnet/flashplayer/articles/stage_video.html :
"The H.264 codec is stage video's best friend; using this will ensure
you get full GPU acceleration from video decoding to rendering. With
this approach, no read-back (sending the data from the GPU to the CPU)
is required to composite the video frames in the display list anymore.
The YUV 4:2:0 <http://en.wikipedia.org/wiki/Chroma_subsampling#4:2:0>
formatted video frames are converted to RGB through a GPU shader
(DirectX9 or OpenGL) and blitted onscreen. As a result, you will see
higher pixel fidelity and some reduction in CPU and memory usage."
then they go on to have a list of limitations finalized by 'In practice,
none of the above restrictions will affect the most common use case,
which is a video player application.'
Y.
> and
> http://www.splitted-desktop.com/static/en/pdf/actu/Linux_with_Gnash.pdf
> might be interesting readings.
>
> Christophe
>
> On Mon, Jun 27, 2011 at 08:32:52AM +0300, Yaniv Kaul wrote:
>> Licensing and patent concerns of x264 aside (see http://mailman.videolan.org/pipermail/x264-devel/2010-July/007508.html),
>> the more I think about it, the more it makes sense to me to have
>> H.264 offload feature in QXL:
>> 1. Many (Flash, HTML5, others) are moving to H.264 - and take
>> advantage of hardware acceleration if such is available, for
>> complete decoding of the stream. If QXL advertises itself as being
>> able to HW accel. the stream, we need to just pick the stream and
>> move it to the client. On the client, we can either download it to
>> the local GPU which will HW accelerate its decoding, or use
>> software-based decoding (x264, for example).
>> 2. Then we can revisit x264 and see if it really provides a
>> noticeable advantage over mjpeg - clearly having one video solution
>> is cleaner and less hassle, and x264 seems to have some nice
>> properties we can use (see http://x264dev.multimedia.cx/archives/249
>> regarding low latency).
>>
>> Both are not trivial tasks, I'm afraid.
>> Y.
>>
>>
>> On 06/25/2011 06:33 PM, John A. Sullivan III wrote:
>>> Hello, all. Andrea Celestino was kind enough to email me off list about
>>> streaming video performance as reducing SPICE bandwidth consumption for
>>> streaming video is his project as a Computer Engineering student. He
>>> agreed that I could repost our conversation to invite input from others
>>> on the list. The (top posted - sorry) thread is below - John
>>>
>>>
>>> Hi, Andrea. No problem with your English. It is certainly better than
>>> my very rusty Italian!
>>>
>>> I know little about video other than as a network engineer and I am not
>>> a programmer but I do know what we need. From that ignorant
>>> perspective, I see several possibilities.
>>>
>>> Video is inherently brutal when it comes to network bandwidth and
>>> processing. There are also different usage scenarios. For example, it
>>> makes sense to stream when the video needs to be played only once and
>>> started immediately. However, if it is going to be viewed multiple
>>> times, the Citrix approach of downloading and playing locally makes more
>>> sense. I've mentioned on the list how we can do this with technology
>>> already in the X2Go project. It is probably the least desirable option
>>> and the least interesting for your project. Of course, we have to
>>> handle all this complexity in a way which does not confuse the end user
>>> who just wants to click on their file or web site and have it work!
>>>
>>> For streaming the video between the SPICE server and client, I think we
>>> also have a couple of options. I don't think there is any magic that
>>> can be done. It is a matter of using the most appropriate codec for the
>>> transmission. The challenge, as already mentioned on the list, is
>>> encoding on the fly. The clients need to decode no matter what but, in
>>> non-SPICE usage, the file is pre-encoded. So, in our selection of
>>> codecs, speed of encoding versus decoding becomes critical. We must
>>> also not only choose one which uses an open source license but one which
>>> is not patent encumbered. I believe that rules out X264. The most
>>> likely candidate is probably VP8 with Theora as an alternative.
>>>
>>> Once a codec (or a selection of codecs) has been chosen, the next
>>> challenge is how to switch. One option is to simply make it a setting
>>> in the SPICE parameters for qemu. The two advantages are that it is the
>>> simplest and it allows the system administrator (who hopefully knows his
>>> system well) to make the balance between CPU utilization and bandwidth.
>>> The disadvantage is that it does not adapt to the users environment and
>>> may require a reboot to change.
>>>
>>> The second option is to select the codec dynamically. Not only would we
>>> select whether to use a lossy or lossless codec based upon the pixel
>>> change rate, but we would then choose the codec based upon the bandwidth
>>> to the client balanced against the bitrate of the video. There is
>>> already precedent for this logic in that SPICE already chooses whether
>>> to use lossy or lossless codecs based upon detected bandwidth. From a
>>> previous post by Marian Krcmarik:
>>>
>>> 'Once client connects to a guest, "spice server" determines client's
>>> bandwidth (look at code for details, I believe It could be more
>>> appropriate and guys are working on it). "jpeg-wan-compression" and
>>> "zlib-glz-wan-compression" are enabled in case the client's bandwidth is
>>> <10Mbps, otherwise It's disabled. When jpeg compression and zlib over
>>> glz compression are enabled then photo-like bitmaps are compressed by
>>> lossy jpeg compression and textual/artificial bitmaps are compressed by
>>> lossy zlib on top of GLZ.'
>>>
>>> SPICE would compare the bandwidth to the bandwidth requirements of the
>>> video and, if the video bandwidth requirements are some percentage less
>>> than the available bandwidth (to leave some of other traffic - I do not
>>> know what that number should be - 80%?), it uses MJPEG for better
>>> quality and lower CPU utilization. If not, it uses VP8 or whatever we
>>> choose. This could be multiple choice depending on bitrate/bandwidth if
>>> that makes sense and not just MJPEG vs VP8.
>>>
>>> This would be the most interesting approach although we should sacrifice
>>> our programming interest to the best solution for the end users. It
>>> does have the advantage of not needing to be tweaked by the admin and
>>> adapting to whether the user is on the end of a 3G cell connection or a
>>> 100 Mbps WAN link.
>>>
>>> As I think about it, we already have precedent for making it the best of
>>> all worlds by using an auto or fixed setting like SPICE does for other
>>> parameters - auto activates the automated routine above and a fixed
>>> value (MJPEG or VP8) specifically locks it to always use the specified
>>> codec. This is probably a good idea in general but would be
>>> particularly helpful in the early stages while we tweak what that magic
>>> percentage is for allowing other traffic, i.e., does the higher
>>> utilization codec kick in at 80%, 60%, 95%.
>>>
>>> Would you mind if I posted this response to the list to see if others
>>> have helpful input? Thanks - John
>>>
>>> On Fri, 2011-06-24 at 11:18 +0200, Andrea Celestino wrote:
>>>> Hi,
>>>>
>>>> first of all I am studying the source coude of spice, because I need
>>>> to understand how spice detects video streaming, how it sends stream
>>>> data to client with socket and how it compress data with mjpeg.
>>>> I have not any idea at the moment, I need to do some tests.
>>>> I am very interested.
>>>> Do you have an idea?
>>>>
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> p.s. sorry for my english :)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2011/6/23 John A. Sullivan III<jsullivan at opensourcedevel.com>
>>>> Certo! Hope I remembered that correctly. I would be extremely
>>>> interested. What did you have in mind? Thanks - John
>>>>
>>>> On Thu, 2011-06-23 at 10:29 +0200, Andrea Celestino wrote:
>>>> > Hi,
>>>> > I have read your message on Spice mailing lists and I am
>>>> interesting
>>>> > in streaming video performance with spice. I am an italian
>>>> student in
>>>> > computer engineering, during this period I am working on
>>>> reducing
>>>> > bandwidth usage with streaming video in Spice. I see that
>>>> you are
>>>> > interested in the same problem. I think that we can
>>>> cooperate. What do
>>>> > you think about it?
>>>>
>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Spice-devel mailing list
>>> Spice-devel at lists.freedesktop.org
>>> http://lists.freedesktop.org/mailman/listinfo/spice-devel
>> _______________________________________________
>> Spice-devel mailing list
>> Spice-devel at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/spice-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/spice-devel/attachments/20110627/2be280fd/attachment.htm>
More information about the Spice-devel
mailing list