[Spice-devel] [PATCH spice-server 00/28] adaptive video streaming
Yonit Halperin
yhalperi at redhat.com
Mon Apr 15 11:54:00 PDT 2013
Hi,
On 04/14/2013 09:37 AM, Alon Levy wrote:
> On Tue, Feb 26, 2013 at 01:03:46PM -0500, Yonit Halperin wrote:
>
> ACK series, sorry for the delay. I have to admit I don't understand the
> first patches as well as I should, but seeing as they have been tested
> and that I would just be delaying them further, I prefer to let you push
> them. I hope to work a bit on testability of streams that doesn't
> require creating a vm, but of course that too is just a note and not
> related to acking this series.
>
Thanks for reviewing.
Anyone care to also review the spice-gtk,spice-common and spice-protocol
patches?
Thanks,
Yonit.
>> Hi,
>>
>> The following patch series introduces adaptive video streaming to spice.
>>
>> Until now, the mjpeg quality was constant (70), and the frame rate was modified
>> according to the rate of frame drops in the server side (a drop occurs when a new frame reaches the server while
>> an older frame is still queued in the pipe). In the client side, the video playback is synchronized according
>> to the audio playback (each audio and video frame holds an mm-time field). The jitter-buffer size in the client
>> was constant as well - 100 ms. When video frames arrive late to the client (i.e., when the audio playback is ahead of them),
>> they are dropped.
>>
>> The adaptive video streaming is implemented by the following heuristic:
>> Given a bit rate, we calculate the best combination of mjpeg quality and frame rate (henceforth, the stream parameters) for this
>> bit rate. In order to decide this combination, we evaluate the encoding size for different jpeg
>> qualities by applying them on successive frames.
>> Every new stream is assigned with an initial bit rate. The bit rate is re-estimated and
>> modified during the stream life time. The bit-rate is modified based on:
>> 1) periodic reports from the client:
>> The client reports includes information about drops and the playback latency.
>> In response to drops, or too short playback latency, we decrease the bit rate.
>> In response to reports that suggest that the client playback is stable with the
>> current configuration, we try to increase the bit-rate.
>> 2) server drops: the bit-rate is decreased when server drops occur.
>>
>> Each time the bit rate changes, the stream parameters are re-evaluated.
>> In addition, we monitor the frames' encoding size, and when there is a change
>> that may allow improving the stream parameters, or alternatively, requires decreasing the
>> quality, we again re-evaluate them.
>>
>> Other changes:
>> --------------
>>
>> Besides the client reports, I also added to the protocol a message that controls the
>> audio playback latency, for allowing better synchronization of the audio and video playback buffering.
>>
>> The roundtrip time is used for estimating the required playback delay. In order to get a more accurate estimation
>> of the roundtrip time I also added an option to measure it periodically instead of just on startup, and
>> take the minimum measurement as estimation.
>>
>> Results
>> -------
>> I compared the video quality of the current spice master, and of the new spice, under different network setups.
>> Spice master was a bit modified for making the comparison more fair: I increased the audio jitter buffer to 200ms (instead of 100),
>> and also included the patch "red_worker: stream agent - fix miscounting of frames".
>> The network setup was emulated using tc.
>>
>> You can find the tests details and the results in a following email.
>>
>> For 5Mpbs and 60ms roundtrip (Test1), in spice-master, more than 70% of the frames that are sent to the client are being dropped, and the video
>> is unwatchable. With new spice, while the average frame rate is about the same, only about 2% of the frames are being dropped by the client.
>> For 2.5Mbps and 60ms (Test2), as expected, things gets worse for spice-master, and the drops rate reaches 90%. For the new spice, it is less then 20%, and
>> the video is watchable.
>>
>> I also tested a setup of 10Mbps with high latency (170ms, Test3). The latency affects the initial bit rate estimation in spice (probably due to the tcp acks overhead).
>> Thus, the stream is started with a bit-rate estimation of less then 1.25Mbps. The adaptive video heuristic gradually converges to a higher bit rate (the column "end-bit-rate"), and
>> the next video stream will be started with the improved bit rate estimation.
>> In Test5 I tested a real environment with a network setup similar to Test3. However, the test are not comparable because in Test5 setup (different server and guest),
>> the basic frame rate (i.e., from the guest to the server) is much smaller (still need to investigate why).
>>
>> In Test4 (20Mbps; <1 ms roundtrip), I evaluated and unlimited setup, i.e., a setup which will allow the best frame rate and jpeg-quality for the stream.
>> With new spice, the capacity of the channel is exploited efficiently. With spice-master, the condition for dropping frames according to the defined fps is too strict,
>> and the observed frame rate is smaller then the maximum possible.
>>
>> Video streaming short-term TODO:
>> ----------------
>> - Implement playback-latency adjustments for spice-gtk gstreamer front-end.
>> - Add vp8 encoding
>> - Solve some problems we have with video identification.
>> - Try to achieve faster convergence to the "right" bit-rate when we start with a wrong estimation.
>>
>> long-term TODO:
>> ---------------
>> video pass-through
>>
>> Regards,
>> Yonit.
>>
>> Yonit Halperin (28):
>> red_worker: stream agent - fix miscounting of frames
>> server/red_worker: streams: moving mjpeg_encoder from Stream to
>> StreamAgent
>> mjpeg_encoder: configure mjpeg quality and frame rate according to a
>> given bit rate
>> mjpeg_encoder: re-configure stream parameters when the frame's
>> encoding size changes
>> mjpeg_encoder: adjust the stream bit rate based on periodic client
>> feedback
>> mjpeg_encoder: modify stream bit rate based on server side pipe
>> congestion
>> mjpeg_encoder: update the client with estimations for the required
>> playback latency
>> mjpeg_encoder: move the control over frame drops to mjpeg_encoder
>> mjpeg_encoder: keep the average observed fps similar to the defined
>> fps
>> mjpeg_encoder: add stream warmup time, in which we avoid server and
>> client drops
>> server: spice_timer_queue
>> server/red_worker: assign timer callbacks to worker_core, using
>> spice_timer_queue
>> red_channel: monitor connection latency using MSG_PING
>> red_worker: stream - update periodically the input frame rate
>> server/red_worker: enable latency monitoring in the display channel
>> red_worker: start using mjpeg_encoder rate control capabilities
>> red_worker: support SPICE_MSGC_DISPLAY_STREAM_REPORT
>> red_worker: notify mjpeg_encoder on server frame drops
>> red_worker: ignoring video frame drops that are not due to pipe
>> congestion
>> dispatcher.h: fix - s/#define MAIN_DISPATCHER_H/#define DISPATCHER_H
>> snd_worker: support sending SPICE_MSG_PLAYBACK_LATENCY
>> reds: support mm_time latency adjustments
>> red_worker: video streams - adjust client playback latency
>> server/red_worker.c: use the bit rate of old streams as a start point
>> for new streams
>> server/red_worker: add an option to supply the bandwidth via env var
>> collect and print video stream statistics
>> red_worker: increase the interval limit for stream frames
>> red_worker: assign mm_time to vga frames
>>
>> server/Makefile.am | 2 +
>> server/dispatcher.h | 6 +-
>> server/inputs_channel.c | 1 +
>> server/main_channel.c | 7 +-
>> server/main_channel.h | 1 +
>> server/main_dispatcher.c | 32 ++
>> server/main_dispatcher.h | 2 +
>> server/mjpeg_encoder.c | 981 ++++++++++++++++++++++++++++++++++++++++++++-
>> server/mjpeg_encoder.h | 70 +++-
>> server/red_channel.c | 228 +++++++++++
>> server/red_channel.h | 18 +
>> server/red_dispatcher.c | 1 +
>> server/red_worker.c | 514 +++++++++++++++++++-----
>> server/reds-private.h | 2 +
>> server/reds.c | 28 +-
>> server/reds.h | 2 +
>> server/smartcard.c | 1 +
>> server/snd_worker.c | 45 +++
>> server/snd_worker.h | 2 +
>> server/spice_timer_queue.c | 268 +++++++++++++
>> server/spice_timer_queue.h | 43 ++
>> server/spicevmc.c | 1 +
>> spice-common | 2 +-
>> 23 files changed, 2148 insertions(+), 109 deletions(-)
>> create mode 100644 server/spice_timer_queue.c
>> create mode 100644 server/spice_timer_queue.h
>>
>> --
>> 1.8.1
>>
>> _______________________________________________
>> Spice-devel mailing list
>> Spice-devel at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/spice-devel
More information about the Spice-devel
mailing list