[Spice-devel] [PATCH spice-server 00/28] adaptive video streaming
Alon Levy
alevy at redhat.com
Wed Apr 17 08:36:28 PDT 2013
> On 04/17/2013 10:47 AM, Alon Levy wrote:
> > On Mon, 2013-04-15 at 14:54 -0400, Yonit Halperin wrote:
> >> Hi,
> >> On 04/14/2013 09:37 AM, Alon Levy wrote:
> >>> On Tue, Feb 26, 2013 at 01:03:46PM -0500, Yonit Halperin wrote:
> >>>
> >>> ACK series, sorry for the delay. I have to admit I don't understand the
> >>> first patches as well as I should, but seeing as they have been tested
> >>> and that I would just be delaying them further, I prefer to let you push
> >>> them. I hope to work a bit on testability of streams that doesn't
> >>> require creating a vm, but of course that too is just a note and not
> >>> related to acking this series.
> >>>
> >> Thanks for reviewing.
> >> Anyone care to also review the spice-gtk,spice-common and spice-protocol
> >> patches?
> >
> > I think it's best I review those too.
> >
> Cool, Thanks.
ACK series including common, protocol & gtk, just sent a couple of small notes but looks good. Pulseaudio corking change (6/8) maybe someone else wants to take a look, I'm not sure about what your comment actually means.
> >>
> >> Thanks,
> >> Yonit.
> >>
> >>>> Hi,
> >>>>
> >>>> The following patch series introduces adaptive video streaming to spice.
> >>>>
> >>>> Until now, the mjpeg quality was constant (70), and the frame rate was
> >>>> modified
> >>>> according to the rate of frame drops in the server side (a drop occurs
> >>>> when a new frame reaches the server while
> >>>> an older frame is still queued in the pipe). In the client side, the
> >>>> video playback is synchronized according
> >>>> to the audio playback (each audio and video frame holds an mm-time
> >>>> field). The jitter-buffer size in the client
> >>>> was constant as well - 100 ms. When video frames arrive late to the
> >>>> client (i.e., when the audio playback is ahead of them),
> >>>> they are dropped.
> >>>>
> >>>> The adaptive video streaming is implemented by the following heuristic:
> >>>> Given a bit rate, we calculate the best combination of mjpeg quality and
> >>>> frame rate (henceforth, the stream parameters) for this
> >>>> bit rate. In order to decide this combination, we evaluate the encoding
> >>>> size for different jpeg
> >>>> qualities by applying them on successive frames.
> >>>> Every new stream is assigned with an initial bit rate. The bit rate is
> >>>> re-estimated and
> >>>> modified during the stream life time. The bit-rate is modified based on:
> >>>> 1) periodic reports from the client:
> >>>> The client reports includes information about drops and the
> >>>> playback latency.
> >>>> In response to drops, or too short playback latency, we decrease
> >>>> the bit rate.
> >>>> In response to reports that suggest that the client playback is
> >>>> stable with the
> >>>> current configuration, we try to increase the bit-rate.
> >>>> 2) server drops: the bit-rate is decreased when server drops occur.
> >>>>
> >>>> Each time the bit rate changes, the stream parameters are re-evaluated.
> >>>> In addition, we monitor the frames' encoding size, and when there is a
> >>>> change
> >>>> that may allow improving the stream parameters, or alternatively,
> >>>> requires decreasing the
> >>>> quality, we again re-evaluate them.
> >>>>
> >>>> Other changes:
> >>>> --------------
> >>>>
> >>>> Besides the client reports, I also added to the protocol a message that
> >>>> controls the
> >>>> audio playback latency, for allowing better synchronization of the audio
> >>>> and video playback buffering.
> >>>>
> >>>> The roundtrip time is used for estimating the required playback delay.
> >>>> In order to get a more accurate estimation
> >>>> of the roundtrip time I also added an option to measure it periodically
> >>>> instead of just on startup, and
> >>>> take the minimum measurement as estimation.
> >>>>
> >>>> Results
> >>>> -------
> >>>> I compared the video quality of the current spice master, and of the new
> >>>> spice, under different network setups.
> >>>> Spice master was a bit modified for making the comparison more fair: I
> >>>> increased the audio jitter buffer to 200ms (instead of 100),
> >>>> and also included the patch "red_worker: stream agent - fix miscounting
> >>>> of frames".
> >>>> The network setup was emulated using tc.
> >>>>
> >>>> You can find the tests details and the results in a following email.
> >>>>
> >>>> For 5Mpbs and 60ms roundtrip (Test1), in spice-master, more than 70% of
> >>>> the frames that are sent to the client are being dropped, and the video
> >>>> is unwatchable. With new spice, while the average frame rate is about
> >>>> the same, only about 2% of the frames are being dropped by the client.
> >>>> For 2.5Mbps and 60ms (Test2), as expected, things gets worse for
> >>>> spice-master, and the drops rate reaches 90%. For the new spice, it is
> >>>> less then 20%, and
> >>>> the video is watchable.
> >>>>
> >>>> I also tested a setup of 10Mbps with high latency (170ms, Test3). The
> >>>> latency affects the initial bit rate estimation in spice (probably due
> >>>> to the tcp acks overhead).
> >>>> Thus, the stream is started with a bit-rate estimation of less then
> >>>> 1.25Mbps. The adaptive video heuristic gradually converges to a higher
> >>>> bit rate (the column "end-bit-rate"), and
> >>>> the next video stream will be started with the improved bit rate
> >>>> estimation.
> >>>> In Test5 I tested a real environment with a network setup similar to
> >>>> Test3. However, the test are not comparable because in Test5 setup
> >>>> (different server and guest),
> >>>> the basic frame rate (i.e., from the guest to the server) is much
> >>>> smaller (still need to investigate why).
> >>>>
> >>>> In Test4 (20Mbps; <1 ms roundtrip), I evaluated and unlimited setup,
> >>>> i.e., a setup which will allow the best frame rate and jpeg-quality for
> >>>> the stream.
> >>>> With new spice, the capacity of the channel is exploited efficiently.
> >>>> With spice-master, the condition for dropping frames according to the
> >>>> defined fps is too strict,
> >>>> and the observed frame rate is smaller then the maximum possible.
> >>>>
> >>>> Video streaming short-term TODO:
> >>>> ----------------
> >>>> - Implement playback-latency adjustments for spice-gtk gstreamer
> >>>> front-end.
> >>>> - Add vp8 encoding
> >>>> - Solve some problems we have with video identification.
> >>>> - Try to achieve faster convergence to the "right" bit-rate when we
> >>>> start with a wrong estimation.
> >>>>
> >>>> long-term TODO:
> >>>> ---------------
> >>>> video pass-through
> >>>>
> >>>> Regards,
> >>>> Yonit.
> >>>>
> >>>> Yonit Halperin (28):
> >>>> red_worker: stream agent - fix miscounting of frames
> >>>> server/red_worker: streams: moving mjpeg_encoder from Stream to
> >>>> StreamAgent
> >>>> mjpeg_encoder: configure mjpeg quality and frame rate according to a
> >>>> given bit rate
> >>>> mjpeg_encoder: re-configure stream parameters when the frame's
> >>>> encoding size changes
> >>>> mjpeg_encoder: adjust the stream bit rate based on periodic client
> >>>> feedback
> >>>> mjpeg_encoder: modify stream bit rate based on server side pipe
> >>>> congestion
> >>>> mjpeg_encoder: update the client with estimations for the required
> >>>> playback latency
> >>>> mjpeg_encoder: move the control over frame drops to mjpeg_encoder
> >>>> mjpeg_encoder: keep the average observed fps similar to the defined
> >>>> fps
> >>>> mjpeg_encoder: add stream warmup time, in which we avoid server and
> >>>> client drops
> >>>> server: spice_timer_queue
> >>>> server/red_worker: assign timer callbacks to worker_core, using
> >>>> spice_timer_queue
> >>>> red_channel: monitor connection latency using MSG_PING
> >>>> red_worker: stream - update periodically the input frame rate
> >>>> server/red_worker: enable latency monitoring in the display channel
> >>>> red_worker: start using mjpeg_encoder rate control capabilities
> >>>> red_worker: support SPICE_MSGC_DISPLAY_STREAM_REPORT
> >>>> red_worker: notify mjpeg_encoder on server frame drops
> >>>> red_worker: ignoring video frame drops that are not due to pipe
> >>>> congestion
> >>>> dispatcher.h: fix - s/#define MAIN_DISPATCHER_H/#define DISPATCHER_H
> >>>> snd_worker: support sending SPICE_MSG_PLAYBACK_LATENCY
> >>>> reds: support mm_time latency adjustments
> >>>> red_worker: video streams - adjust client playback latency
> >>>> server/red_worker.c: use the bit rate of old streams as a start
> >>>> point
> >>>> for new streams
> >>>> server/red_worker: add an option to supply the bandwidth via env var
> >>>> collect and print video stream statistics
> >>>> red_worker: increase the interval limit for stream frames
> >>>> red_worker: assign mm_time to vga frames
> >>>>
> >>>> server/Makefile.am | 2 +
> >>>> server/dispatcher.h | 6 +-
> >>>> server/inputs_channel.c | 1 +
> >>>> server/main_channel.c | 7 +-
> >>>> server/main_channel.h | 1 +
> >>>> server/main_dispatcher.c | 32 ++
> >>>> server/main_dispatcher.h | 2 +
> >>>> server/mjpeg_encoder.c | 981
> >>>> ++++++++++++++++++++++++++++++++++++++++++++-
> >>>> server/mjpeg_encoder.h | 70 +++-
> >>>> server/red_channel.c | 228 +++++++++++
> >>>> server/red_channel.h | 18 +
> >>>> server/red_dispatcher.c | 1 +
> >>>> server/red_worker.c | 514 +++++++++++++++++++-----
> >>>> server/reds-private.h | 2 +
> >>>> server/reds.c | 28 +-
> >>>> server/reds.h | 2 +
> >>>> server/smartcard.c | 1 +
> >>>> server/snd_worker.c | 45 +++
> >>>> server/snd_worker.h | 2 +
> >>>> server/spice_timer_queue.c | 268 +++++++++++++
> >>>> server/spice_timer_queue.h | 43 ++
> >>>> server/spicevmc.c | 1 +
> >>>> spice-common | 2 +-
> >>>> 23 files changed, 2148 insertions(+), 109 deletions(-)
> >>>> create mode 100644 server/spice_timer_queue.c
> >>>> create mode 100644 server/spice_timer_queue.h
> >>>>
> >>>> --
> >>>> 1.8.1
> >>>>
> >>>> _______________________________________________
> >>>> Spice-devel mailing list
> >>>> Spice-devel at lists.freedesktop.org
> >>>> http://lists.freedesktop.org/mailman/listinfo/spice-devel
> >>
> >
> >
>
>
More information about the Spice-devel
mailing list