[Spice-devel] [PATCH spice-server 00/28] adaptive video streaming
Yonit Halperin
yhalperi at redhat.com
Thu Mar 7 12:52:04 PST 2013
Hi,
On 03/03/2013 07:50 AM, Andrew Cathrow wrote:
>
>
> ----- Original Message -----
>> From: "Yonit Halperin" <yhalperi at redhat.com>
>> To: spice-devel at freedesktop.org
>> Sent: Tuesday, February 26, 2013 1:03:46 PM
>> Subject: [Spice-devel] [PATCH spice-server 00/28] adaptive video streaming
>>
>> Hi,
>>
>> The following patch series introduces adaptive video streaming to
>> spice.
>>
>> Until now, the mjpeg quality was constant (70), and the frame rate
>> was modified
>> according to the rate of frame drops in the server side (a drop
>> occurs when a new frame reaches the server while
>> an older frame is still queued in the pipe). In the client side, the
>> video playback is synchronized according
>> to the audio playback (each audio and video frame holds an mm-time
>> field). The jitter-buffer size in the client
>> was constant as well - 100 ms. When video frames arrive late to the
>> client (i.e., when the audio playback is ahead of them),
>> they are dropped.
>>
>> The adaptive video streaming is implemented by the following
>> heuristic:
>> Given a bit rate, we calculate the best combination of mjpeg quality
>> and frame rate (henceforth, the stream parameters) for this
>> bit rate. In order to decide this combination, we evaluate the
>> encoding size for different jpeg
>> qualities by applying them on successive frames.
>> Every new stream is assigned with an initial bit rate. The bit rate
>> is re-estimated and
>> modified during the stream life time. The bit-rate is modified based
>> on:
>> 1) periodic reports from the client:
>> The client reports includes information about drops and the
>> playback latency.
>> In response to drops, or too short playback latency, we decrease
>> the bit rate.
>> In response to reports that suggest that the client playback is
>> stable with the
>> current configuration, we try to increase the bit-rate.
>> 2) server drops: the bit-rate is decreased when server drops occur.
>>
>> Each time the bit rate changes, the stream parameters are
>> re-evaluated.
>> In addition, we monitor the frames' encoding size, and when there is
>> a change
>> that may allow improving the stream parameters, or alternatively,
>> requires decreasing the
>> quality, we again re-evaluate them.
>
>
> What kind of metrics do we expose - how can I get the bitrate back on the client and/or server?
>
The server side estimates the latency, and an optimal bit rate for the
video playback, which is dynamically modified throughout the playback.
This measurements are kept on the server (they are documented in the log
file if you run with the highest debug level).
>>
>> Other changes:
>> --------------
>>
>> Besides the client reports, I also added to the protocol a message
>> that controls the
>> audio playback latency, for allowing better synchronization of the
>> audio and video playback buffering.
>>
>> The roundtrip time is used for estimating the required playback
>> delay. In order to get a more accurate estimation
>> of the roundtrip time I also added an option to measure it
>> periodically instead of just on startup, and
>> take the minimum measurement as estimation.
>>
>> Results
>> -------
>> I compared the video quality of the current spice master, and of the
>> new spice, under different network setups.
>> Spice master was a bit modified for making the comparison more fair:
>> I increased the audio jitter buffer to 200ms (instead of 100),
>> and also included the patch "red_worker: stream agent - fix
>> miscounting of frames".
>> The network setup was emulated using tc.
>>
>> You can find the tests details and the results in a following email.
>>
>> For 5Mpbs and 60ms roundtrip (Test1), in spice-master, more than 70%
>> of the frames that are sent to the client are being dropped, and the
>> video
>> is unwatchable. With new spice, while the average frame rate is about
>> the same, only about 2% of the frames are being dropped by the
>> client.
>> For 2.5Mbps and 60ms (Test2), as expected, things gets worse for
>> spice-master, and the drops rate reaches 90%. For the new spice, it
>> is less then 20%, and
>> the video is watchable.
>>
>> I also tested a setup of 10Mbps with high latency (170ms, Test3). The
>> latency affects the initial bit rate estimation in spice (probably
>> due to the tcp acks overhead).
>> Thus, the stream is started with a bit-rate estimation of less then
>> 1.25Mbps. The adaptive video heuristic gradually converges to a
>> higher bit rate (the column "end-bit-rate"), and
>> the next video stream will be started with the improved bit rate
>> estimation.
>> In Test5 I tested a real environment with a network setup similar to
>> Test3. However, the test are not comparable because in Test5 setup
>> (different server and guest),
>> the basic frame rate (i.e., from the guest to the server) is much
>> smaller (still need to investigate why).
>>
>> In Test4 (20Mbps; <1 ms roundtrip), I evaluated and unlimited setup,
>> i.e., a setup which will allow the best frame rate and jpeg-quality
>> for the stream.
>> With new spice, the capacity of the channel is exploited efficiently.
>> With spice-master, the condition for dropping frames according to
>> the defined fps is too strict,
>> and the observed frame rate is smaller then the maximum possible.
>>
>> Video streaming short-term TODO:
>> ----------------
>> - Implement playback-latency adjustments for spice-gtk gstreamer
>> front-end.
>> - Add vp8 encoding
>> - Solve some problems we have with video identification.
>> - Try to achieve faster convergence to the "right" bit-rate when we
>> start with a wrong estimation.
>>
>> long-term TODO:
>> ---------------
>> video pass-through
>>
>> Regards,
>> Yonit.
>>
>> Yonit Halperin (28):
>> red_worker: stream agent - fix miscounting of frames
>> server/red_worker: streams: moving mjpeg_encoder from Stream to
>> StreamAgent
>> mjpeg_encoder: configure mjpeg quality and frame rate according to
>> a
>> given bit rate
>> mjpeg_encoder: re-configure stream parameters when the frame's
>> encoding size changes
>> mjpeg_encoder: adjust the stream bit rate based on periodic client
>> feedback
>> mjpeg_encoder: modify stream bit rate based on server side pipe
>> congestion
>> mjpeg_encoder: update the client with estimations for the required
>> playback latency
>> mjpeg_encoder: move the control over frame drops to mjpeg_encoder
>> mjpeg_encoder: keep the average observed fps similar to the defined
>> fps
>> mjpeg_encoder: add stream warmup time, in which we avoid server and
>> client drops
>> server: spice_timer_queue
>> server/red_worker: assign timer callbacks to worker_core, using
>> spice_timer_queue
>> red_channel: monitor connection latency using MSG_PING
>> red_worker: stream - update periodically the input frame rate
>> server/red_worker: enable latency monitoring in the display channel
>> red_worker: start using mjpeg_encoder rate control capabilities
>> red_worker: support SPICE_MSGC_DISPLAY_STREAM_REPORT
>> red_worker: notify mjpeg_encoder on server frame drops
>> red_worker: ignoring video frame drops that are not due to pipe
>> congestion
>> dispatcher.h: fix - s/#define MAIN_DISPATCHER_H/#define
>> DISPATCHER_H
>> snd_worker: support sending SPICE_MSG_PLAYBACK_LATENCY
>> reds: support mm_time latency adjustments
>> red_worker: video streams - adjust client playback latency
>> server/red_worker.c: use the bit rate of old streams as a start
>> point
>> for new streams
>> server/red_worker: add an option to supply the bandwidth via env
>> var
>> collect and print video stream statistics
>> red_worker: increase the interval limit for stream frames
>> red_worker: assign mm_time to vga frames
>>
>> server/Makefile.am | 2 +
>> server/dispatcher.h | 6 +-
>> server/inputs_channel.c | 1 +
>> server/main_channel.c | 7 +-
>> server/main_channel.h | 1 +
>> server/main_dispatcher.c | 32 ++
>> server/main_dispatcher.h | 2 +
>> server/mjpeg_encoder.c | 981
>> ++++++++++++++++++++++++++++++++++++++++++++-
>> server/mjpeg_encoder.h | 70 +++-
>> server/red_channel.c | 228 +++++++++++
>> server/red_channel.h | 18 +
>> server/red_dispatcher.c | 1 +
>> server/red_worker.c | 514 +++++++++++++++++++-----
>> server/reds-private.h | 2 +
>> server/reds.c | 28 +-
>> server/reds.h | 2 +
>> server/smartcard.c | 1 +
>> server/snd_worker.c | 45 +++
>> server/snd_worker.h | 2 +
>> server/spice_timer_queue.c | 268 +++++++++++++
>> server/spice_timer_queue.h | 43 ++
>> server/spicevmc.c | 1 +
>> spice-common | 2 +-
>> 23 files changed, 2148 insertions(+), 109 deletions(-)
>> create mode 100644 server/spice_timer_queue.c
>> create mode 100644 server/spice_timer_queue.h
>>
>> --
>> 1.8.1
>>
>> _______________________________________________
>> Spice-devel mailing list
>> Spice-devel at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/spice-devel
>>
More information about the Spice-devel
mailing list