[Spice-devel] [PATCH spice-server 00/28] adaptive video streaming

Tue Feb 26 10:03:46 PST 2013

Hi,

The following patch series introduces adaptive video streaming to spice.

Until now, the mjpeg quality was constant (70), and the frame rate was modified
according to the rate of frame drops in the server side (a drop occurs when a new frame reaches the server while
an older frame is still queued in the pipe). In the client side, the video playback is synchronized according
to the audio playback (each audio and video frame holds an mm-time field). The jitter-buffer size in the client
was constant as well - 100 ms. When video frames arrive late to the client (i.e., when the audio playback is ahead of them),
they are dropped.

The adaptive video streaming is implemented by the following heuristic:
Given a bit rate, we calculate the best combination of mjpeg quality and frame rate (henceforth, the stream parameters) for this
bit rate. In order to decide this combination, we evaluate the encoding size for different jpeg
qualities by applying them on successive frames.
Every new stream is assigned with an initial bit rate. The bit rate is re-estimated and
modified during the stream life time. The bit-rate is modified based on:
1) periodic reports from the client:
   The client reports includes information about drops and the playback latency.
   In response to drops, or too short playback latency, we decrease the bit rate.
   In response to reports that suggest that the client playback is stable with the
   current configuration, we try to increase the bit-rate.
2) server drops: the bit-rate is decreased when server drops occur.

Each time the bit rate changes, the stream parameters are re-evaluated.
In addition, we monitor the frames' encoding size, and when there is a change
that may allow improving the stream parameters, or alternatively, requires decreasing the
quality, we again re-evaluate them.

Other changes:
--------------

Besides the client reports, I also added to the protocol a message that controls the
audio playback latency, for allowing better synchronization of the audio and video playback buffering.

The roundtrip time is used for estimating the required playback delay. In order to get a more accurate estimation
of the roundtrip time I also added an option to measure it periodically instead of just on startup, and
take the minimum measurement as estimation.

Results
-------
I compared the video quality of the current spice master, and of the new spice, under different network setups.
Spice master was a bit modified for making the comparison more fair: I increased the audio jitter buffer to 200ms (instead of 100),
and also included the patch "red_worker: stream agent - fix miscounting of frames".
The network setup was emulated using tc.

You can find the tests details and the results in a following email.

For 5Mpbs and 60ms roundtrip (Test1), in spice-master, more than 70% of the frames that are sent to the client are being dropped, and the video
is unwatchable. With new spice, while the average frame rate is about the same, only about 2% of the frames are being dropped by the client.
For 2.5Mbps and 60ms (Test2), as expected, things gets worse for spice-master, and the drops rate reaches 90%. For the new spice, it is less then 20%, and
the video is watchable.

I also tested a setup of 10Mbps with high latency (170ms, Test3). The latency affects the initial bit rate estimation in spice (probably due to the tcp acks overhead).
Thus, the stream is started with a bit-rate estimation of less then 1.25Mbps. The adaptive video heuristic gradually converges to a higher bit rate (the column "end-bit-rate"), and
the next video stream will be started with the improved bit rate estimation.
In Test5 I tested a real environment with a network setup similar to Test3. However, the test are not comparable because in Test5 setup (different server and guest),
the basic frame rate (i.e., from the guest to the server) is much smaller (still need to investigate why).

In Test4 (20Mbps; <1 ms roundtrip), I evaluated and unlimited setup, i.e., a setup which will allow the best frame rate and jpeg-quality for the stream.
With new spice, the capacity of the channel is exploited efficiently. With spice-master, the condition for dropping frames according to the defined fps is too strict,
and the observed frame rate is smaller then the maximum possible.

Video streaming short-term TODO:
----------------
- Implement playback-latency adjustments for spice-gtk gstreamer front-end.
- Add vp8 encoding
- Solve some problems we have with video identification.
- Try to achieve faster convergence to the "right" bit-rate when we start with a wrong estimation.

long-term TODO:
---------------
video pass-through

Regards,
Yonit.

Yonit Halperin (28):
  red_worker: stream agent - fix miscounting of frames
  server/red_worker: streams: moving mjpeg_encoder from Stream to
    StreamAgent
  mjpeg_encoder: configure mjpeg quality and frame rate according to a
    given bit rate
  mjpeg_encoder: re-configure stream parameters when the frame's
    encoding size changes
  mjpeg_encoder: adjust the stream bit rate based on periodic client
    feedback
  mjpeg_encoder: modify stream bit rate based on server side pipe
    congestion
  mjpeg_encoder: update the client with estimations for the required
    playback latency
  mjpeg_encoder: move the control over frame drops to mjpeg_encoder
  mjpeg_encoder: keep the average observed fps similar to the defined
    fps
  mjpeg_encoder: add stream warmup time, in which we avoid server and
    client drops
  server: spice_timer_queue
  server/red_worker: assign timer callbacks to worker_core, using
    spice_timer_queue
  red_channel: monitor connection latency using MSG_PING
  red_worker: stream - update periodically the input frame rate
  server/red_worker: enable latency monitoring in the display channel
  red_worker: start using mjpeg_encoder rate control capabilities
  red_worker: support SPICE_MSGC_DISPLAY_STREAM_REPORT
  red_worker: notify mjpeg_encoder on server frame drops
  red_worker: ignoring video frame drops that are not due to pipe
    congestion
  dispatcher.h: fix - s/#define MAIN_DISPATCHER_H/#define DISPATCHER_H
  snd_worker: support sending SPICE_MSG_PLAYBACK_LATENCY
  reds: support mm_time latency adjustments
  red_worker: video streams - adjust client playback latency
  server/red_worker.c: use the bit rate of old streams as a start point
    for new streams
  server/red_worker: add an option to supply the bandwidth via env var
  collect and print video stream statistics
  red_worker: increase the interval limit for stream frames
  red_worker: assign mm_time to vga frames

 server/Makefile.am         |   2 +
 server/dispatcher.h        |   6 +-
 server/inputs_channel.c    |   1 +
 server/main_channel.c      |   7 +-
 server/main_channel.h      |   1 +
 server/main_dispatcher.c   |  32 ++
 server/main_dispatcher.h   |   2 +
 server/mjpeg_encoder.c     | 981 ++++++++++++++++++++++++++++++++++++++++++++-
 server/mjpeg_encoder.h     |  70 +++-
 server/red_channel.c       | 228 +++++++++++
 server/red_channel.h       |  18 +
 server/red_dispatcher.c    |   1 +
 server/red_worker.c        | 514 +++++++++++++++++++-----
 server/reds-private.h      |   2 +
 server/reds.c              |  28 +-
 server/reds.h              |   2 +
 server/smartcard.c         |   1 +
 server/snd_worker.c        |  45 +++
 server/snd_worker.h        |   2 +
 server/spice_timer_queue.c | 268 +++++++++++++
 server/spice_timer_queue.h |  43 ++
 server/spicevmc.c          |   1 +
 spice-common               |   2 +-
 23 files changed, 2148 insertions(+), 109 deletions(-)
 create mode 100644 server/spice_timer_queue.c
 create mode 100644 server/spice_timer_queue.h

-- 
1.8.1