[Spice-devel] [PATCH spice-server 00/28] adaptive video streaming

[Yonit Halperin]

Hi,

On 02/26/2013 01:40 PM, Marc-André Lureau wrote:
> Hi
>
> ----- Mensaje original -----
>> The adaptive video streaming is implemented by the following
>> heuristic:
>> Given a bit rate, we calculate the best combination of mjpeg quality
>> and frame rate (henceforth, the stream parameters) for this
>> bit rate. In order to decide this combination, we evaluate the
>> encoding size for different jpeg
>> qualities by applying them on successive frames.
>
> But no downscaling?
Good point. However, the default libjpeg sampling factors values are 
already 2 for luminance components and 1 for chrominance components 
(both horizontal and vertical), while 4 is the highest value, and larger 
factor means higher resolution. It is worth trying decreasing the 
luminance to 1.

>
>> Every new stream is assigned with an initial bit rate. The bit rate
>> is re-estimated and
>> modified during the stream life time. The bit-rate is modified based
>> on:
>> 1) periodic reports from the client:
>>     The client reports includes information about drops and the
>>     playback latency.
>>     In response to drops, or too short playback latency, we decrease
>>     the bit rate.
>>     In response to reports that suggest that the client playback is
>>     stable with the
>>     current configuration, we try to increase the bit-rate.
>
> How is this going to work with multiple client?
Each client have a separate rate-controller and a different stream. Ages 
ago, we (alon an I) talked about clustering clients according to their 
network setup, and encode each message only once per cluster. I think 
that when we have such mechanism, we can use one rate controller per 
cluster, and then we can base our estimations on the "weakest" client in 
the cluster.
>
>> 2) server drops: the bit-rate is decreased when server drops occur.
>> Each time the bit rate changes, the stream parameters are
>> re-evaluated.
>> In addition, we monitor the frames' encoding size, and when there is
>> a change
>> that may allow improving the stream parameters, or alternatively,
>> requires decreasing the
>> quality, we again re-evaluate them.
>>
>> Results
>> -------
>> I compared the video quality of the current spice master, and of the
>> new spice, under different network setups.
>> Spice master was a bit modified for making the comparison more fair:
>> I increased the audio jitter buffer to 200ms (instead of 100),
>> and also included the patch "red_worker: stream agent - fix
>> miscounting of frames".
>> The network setup was emulated using tc.
>>
>> You can find the tests details and the results in a following email.
>>
>> For 5Mpbs and 60ms roundtrip (Test1), in spice-master, more than 70%
>> of the frames that are sent to the client are being dropped, and the
>> video
>> is unwatchable. With new spice, while the average frame rate is about
>> the same, only about 2% of the frames are being dropped by the
>> client.
>> For 2.5Mbps and 60ms (Test2), as expected, things gets worse for
>> spice-master, and the drops rate reaches 90%. For the new spice, it
>> is less then 20%, and
>> the video is watchable.
>
> Really nice!
>
>> I also tested a setup of 10Mbps with high latency (170ms, Test3). The
>> latency affects the initial bit rate estimation in spice (probably
>> due to the tcp acks overhead).
>> Thus, the stream is started with a bit-rate estimation of less then
>> 1.25Mbps. The adaptive video heuristic gradually converges to a
>> higher bit rate (the column "end-bit-rate"), and
>> the next video stream will be started with the improved bit rate
>> estimation.
>> In Test5 I tested a real environment with a network setup similar to
>> Test3. However, the test are not comparable because in Test5 setup
>> (different server and guest),
>> the basic frame rate (i.e., from the guest to the server) is much
>> smaller (still need to investigate why).
>>
>> In Test4 (20Mbps; <1 ms roundtrip), I evaluated and unlimited setup,
>> i.e., a setup which will allow the best frame rate and jpeg-quality
>> for the stream.
>> With new spice, the capacity of the channel is exploited efficiently.
>> With spice-master, the condition for dropping frames according to
>> the defined fps is too strict,
>> and the observed frame rate is smaller then the maximum possible.
>>
>> Video streaming short-term TODO:
>> ----------------
>> - Implement playback-latency adjustments for spice-gtk gstreamer
>> front-end.
>
> Shouldn't be too hard. Is it working fine with pulse backend already? Perhaps that will be answer with another patch series :)
>
I tried using setting the pulse prebuf and tlength, but it didn't work 
so well, so instead, I corked and uncorked the playback explicitly. Once 
this series is complete and pushed, I'll work on gstreamer as well.

>> - Add vp8 encoding
>
> Video codec is really bad, it will really take a lot of CPU, unless it can be done on GPU?
>
I haven't tried it yet, is it even worse than the libjpeg-turbo 
performance? For high resolution videos, the cpu usage for libjpeg-turbo 
is very high.
About hardware acceleration (at least for decoding): 
http://en.wikipedia.org/wiki/WebM#Hardware
>> - Solve some problems we have with video identification.
>
> Do you mean detecting video regions?
>
Yes, it's not as good as it used to be, mainly for flash and html5. 
Still need to track down all the reasons.

>> - Try to achieve faster convergence to the "right" bit-rate when we
>> start with a wrong estimation.
>
> Tricky question, voip application have similar problems, as well as adaptive bandwidth clients (SS/HLS/dash). RTP uses TFRC for smoothing bandwidth, could be an idea to discuss with people on #farstream for example.
Thanks for the references!
>
>> long-term TODO:
>> ---------------
>> video pass-through
>
> This one is hard to prioritize, on the one hand the benefits for server and quality is huge, otoh it won't probably just work in many cases, especially if the final composition isn't done on client side.
>
Yes, it is not that trivial: several guest applications to support; 
dependency on the client platform; proprietary codecs; etc.

Cheers,
Yonit.