[Spice-devel] [PATCH spice-server 00/28] adaptive video streaming

Sun Mar 3 04:50:52 PST 2013

----- Original Message -----
> From: "Yonit Halperin" <yhalperi at redhat.com>
> To: spice-devel at freedesktop.org
> Sent: Tuesday, February 26, 2013 1:03:46 PM
> Subject: [Spice-devel] [PATCH spice-server 00/28] adaptive video streaming
> 
> Hi,
> 
> The following patch series introduces adaptive video streaming to
> spice.
> 
> Until now, the mjpeg quality was constant (70), and the frame rate
> was modified
> according to the rate of frame drops in the server side (a drop
> occurs when a new frame reaches the server while
> an older frame is still queued in the pipe). In the client side, the
> video playback is synchronized according
> to the audio playback (each audio and video frame holds an mm-time
> field). The jitter-buffer size in the client
> was constant as well - 100 ms. When video frames arrive late to the
> client (i.e., when the audio playback is ahead of them),
> they are dropped.
> 
> The adaptive video streaming is implemented by the following
> heuristic:
> Given a bit rate, we calculate the best combination of mjpeg quality
> and frame rate (henceforth, the stream parameters) for this
> bit rate. In order to decide this combination, we evaluate the
> encoding size for different jpeg
> qualities by applying them on successive frames.
> Every new stream is assigned with an initial bit rate. The bit rate
> is re-estimated and
> modified during the stream life time. The bit-rate is modified based
> on:
> 1) periodic reports from the client:
>    The client reports includes information about drops and the
>    playback latency.
>    In response to drops, or too short playback latency, we decrease
>    the bit rate.
>    In response to reports that suggest that the client playback is
>    stable with the
>    current configuration, we try to increase the bit-rate.
> 2) server drops: the bit-rate is decreased when server drops occur.
> 
> Each time the bit rate changes, the stream parameters are
> re-evaluated.
> In addition, we monitor the frames' encoding size, and when there is
> a change
> that may allow improving the stream parameters, or alternatively,
> requires decreasing the
> quality, we again re-evaluate them.

What kind of metrics do we expose - how can I get the bitrate back on the client and/or server?

> 
> Other changes:
> --------------
> 
> Besides the client reports, I also added to the protocol a message
> that controls the
> audio playback latency, for allowing better synchronization of the
> audio and video playback buffering.
> 
> The roundtrip time is used for estimating the required playback
> delay. In order to get a more accurate estimation
> of the roundtrip time I also added an option to measure it
> periodically instead of just on startup, and
> take the minimum measurement as estimation.
> 
> Results
> -------
> I compared the video quality of the current spice master, and of the
> new spice, under different network setups.
> Spice master was a bit modified for making the comparison more fair:
> I increased the audio jitter buffer to 200ms (instead of 100),
> and also included the patch "red_worker: stream agent - fix
> miscounting of frames".
> The network setup was emulated using tc.
> 
> You can find the tests details and the results in a following email.
> 
> For 5Mpbs and 60ms roundtrip (Test1), in spice-master, more than 70%
> of the frames that are sent to the client are being dropped, and the
> video
> is unwatchable. With new spice, while the average frame rate is about
> the same, only about 2% of the frames are being dropped by the
> client.
> For 2.5Mbps and 60ms (Test2), as expected, things gets worse for
> spice-master, and the drops rate reaches 90%. For the new spice, it
> is less then 20%, and
> the video is watchable.
> 
> I also tested a setup of 10Mbps with high latency (170ms, Test3). The
> latency affects the initial bit rate estimation in spice (probably
> due to the tcp acks overhead).
> Thus, the stream is started with a bit-rate estimation of less then
> 1.25Mbps. The adaptive video heuristic gradually converges to a
> higher bit rate (the column "end-bit-rate"), and
> the next video stream will be started with the improved bit rate
> estimation.
> In Test5 I tested a real environment with a network setup similar to
> Test3. However, the test are not comparable because in Test5 setup
> (different server and guest),
> the basic frame rate (i.e., from the guest to the server) is much
> smaller (still need to investigate why).
> 
> In Test4 (20Mbps; <1 ms roundtrip), I evaluated and unlimited setup,
> i.e., a setup which will allow the best frame rate and jpeg-quality
> for the stream.
> With new spice, the capacity of the channel is exploited efficiently.
> With spice-master, the condition for dropping frames according to
> the defined fps is too strict,
> and the observed frame rate is smaller then the maximum possible.
> 
> Video streaming short-term TODO:
> ----------------
> - Implement playback-latency adjustments for spice-gtk gstreamer
> front-end.
> - Add vp8 encoding
> - Solve some problems we have with video identification.
> - Try to achieve faster convergence to the "right" bit-rate when we
> start with a wrong estimation.
> 
> long-term TODO:
> ---------------
> video pass-through
> 
> Regards,
> Yonit.
> 
> Yonit Halperin (28):
>   red_worker: stream agent - fix miscounting of frames
>   server/red_worker: streams: moving mjpeg_encoder from Stream to
>     StreamAgent
>   mjpeg_encoder: configure mjpeg quality and frame rate according to
>   a
>     given bit rate
>   mjpeg_encoder: re-configure stream parameters when the frame's
>     encoding size changes
>   mjpeg_encoder: adjust the stream bit rate based on periodic client
>     feedback
>   mjpeg_encoder: modify stream bit rate based on server side pipe
>     congestion
>   mjpeg_encoder: update the client with estimations for the required
>     playback latency
>   mjpeg_encoder: move the control over frame drops to mjpeg_encoder
>   mjpeg_encoder: keep the average observed fps similar to the defined
>     fps
>   mjpeg_encoder: add stream warmup time, in which we avoid server and
>     client drops
>   server: spice_timer_queue
>   server/red_worker: assign timer callbacks to worker_core, using
>     spice_timer_queue
>   red_channel: monitor connection latency using MSG_PING
>   red_worker: stream - update periodically the input frame rate
>   server/red_worker: enable latency monitoring in the display channel
>   red_worker: start using mjpeg_encoder rate control capabilities
>   red_worker: support SPICE_MSGC_DISPLAY_STREAM_REPORT
>   red_worker: notify mjpeg_encoder on server frame drops
>   red_worker: ignoring video frame drops that are not due to pipe
>     congestion
>   dispatcher.h: fix - s/#define MAIN_DISPATCHER_H/#define
>   DISPATCHER_H
>   snd_worker: support sending SPICE_MSG_PLAYBACK_LATENCY
>   reds: support mm_time latency adjustments
>   red_worker: video streams - adjust client playback latency
>   server/red_worker.c: use the bit rate of old streams as a start
>   point
>     for new streams
>   server/red_worker: add an option to supply the bandwidth via env
>   var
>   collect and print video stream statistics
>   red_worker: increase the interval limit for stream frames
>   red_worker: assign mm_time to vga frames
> 
>  server/Makefile.am         |   2 +
>  server/dispatcher.h        |   6 +-
>  server/inputs_channel.c    |   1 +
>  server/main_channel.c      |   7 +-
>  server/main_channel.h      |   1 +
>  server/main_dispatcher.c   |  32 ++
>  server/main_dispatcher.h   |   2 +
>  server/mjpeg_encoder.c     | 981
>  ++++++++++++++++++++++++++++++++++++++++++++-
>  server/mjpeg_encoder.h     |  70 +++-
>  server/red_channel.c       | 228 +++++++++++
>  server/red_channel.h       |  18 +
>  server/red_dispatcher.c    |   1 +
>  server/red_worker.c        | 514 +++++++++++++++++++-----
>  server/reds-private.h      |   2 +
>  server/reds.c              |  28 +-
>  server/reds.h              |   2 +
>  server/smartcard.c         |   1 +
>  server/snd_worker.c        |  45 +++
>  server/snd_worker.h        |   2 +
>  server/spice_timer_queue.c | 268 +++++++++++++
>  server/spice_timer_queue.h |  43 ++
>  server/spicevmc.c          |   1 +
>  spice-common               |   2 +-
>  23 files changed, 2148 insertions(+), 109 deletions(-)
>  create mode 100644 server/spice_timer_queue.c
>  create mode 100644 server/spice_timer_queue.h
> 
> --
> 1.8.1
> 
> _______________________________________________
> Spice-devel mailing list
> Spice-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/spice-devel
> 