[Spice-devel] [spice v14 00/29] Add GStreamer support for video streaming

[Francois Gouget]

This patch series adds support for using GStreamer to encode and 
decode the video streams, adding support for VP8 and h264 codecs.

As before the patches can also be grabbed from the gst branch of the
repositories below:

spice:          https://github.com/fgouget/spice
spice-gtk:      https://github.com/fgouget/spice-gtk
xf86-video-qxl: https://github.com/fgouget/xf86-video-qxl
spice-protocol: https://github.com/fgouget/spice-protocol

See also gst-sync for the old spice-gtk code. (there's also 'extras'
branches with more experimental/future patches for the curious)

Note also that I added gst-v10, gst-v11, gst-v12 and gst-v13 branches to 
help comparing these revisions.

Note that to test GStreamer support with QEMU you probably need to grab 
the patch below and set the video-codecs option to something like 
gstreamer:h264 or gstreamer:vp8.

https://lists.freedesktop.org/archives/spice-devel/2015-May/019771.html


Changes from v13:

Rebased + some changes suggested by Christophe Fergeau.


Changes from v12:

This round focused on performance and reliability in various special 
cases such as low bandwidth operation.

 * The server no longer rebuilds the pipeline from scratch every time 
   the frame size or bitrate changes. The trick to it is to transition 
   to the NULL state (instead of the PAUSED one as before).
 * I also unified configure_pipeline() and reconfigure_pipeline() by 
   moving all the one-time configuration operations to 
   create_pipeline() and tracking the configuration changes to apply.
 * The server can now also apply bitrate changes directly if the codec 
   advertises it can be changed in the PLAYING state. This is the case 
   for the x264enc codec.
 * The server can now detect when the GStreamer pipeline error occurs 
   instead of getting stuck waiting for the compressed buffer.
 * I fixed the value for the error-resilient vp8enc option.
 * I removed the resize-allowed=true vp8enc setting as tests showed the 
   decoder was sometimes not restoring a full-size frame.
 * I removed the 'frame' SpiceGstVideoBuffer field as it was not used.
 * push_raw_frame() now avoids allocating memory for chunk_copy() if 
   zero_copy() was able to take care of the whole frame.

 * The client no longer rebuilds or even reconfigure the pipeline from 
   scratch every time the frame size changes. It turns out that the 
   decoders don't care (unlike the encoders).
 * I have protected the PTS calculation against underflows. This fixes 
   freezes that could occasionally happen with negative latencies near 
   the start of a stream (i.e. in low bandwidth cases).
 * I renamed SpiceFrameMeta to just SpiceFrame since it also contains 
   the sample once the frame has been decoded, and is thus not just 
   metadata. I also cleaned up the naming of the corresponding queues.
 * The GStreamer pipeline sometimes returns multiple buffers with the 
   same PTS. The first buffer would remove its SpiceFrame from the 
   decoding_queue so that the next one would not find a match, thus 
   clearing the decoding_queue in the process. As a result the next 
   buffer would not find a match either, again clearing the 
   decoding_queue in the process, etc. So buffers that have an unknown 
   PTS no longer clear the decoding_queue.
 * When frames are late for display the client now tries to display the 
   least out of date one in the display_queue so there are at least 
   some screen updates.
 * I removed the qos=true appsink option. It's not clear that it was 
   helping so I decided to remove it until there is more evidence that 
   it is needed.
 * I renamed stream_dropped_frame() to 
   stream_dropped_frame_on_playback() to make it clearer that it's 
   unrelated to the stream report data (which is based on arrival time).


Besides rebases, the changes from v11 are limited to spice-gtk. 
Specifically they fix two compatibility issues:
 * v11 used a GAsyncQueue and relied on g_async_queue_push_front() which 
   is only available in GLib >= 2.46 which is unfortunately not 
   available in Debian 8 and presumably on many other Linux 
   distributions.
 * It also associated the original network message with the 
   corresponding GStreamer buffer using the GstMeta API. This is 
   necessary in order to know where to display the decoded frame as this 
   could change from one frame to the next. However some GStreamer 
   elements have (refcounting) bugs that caused the metadata to be lost 
   in the pipeline. I had a workaround in place that worked on Debian 
   Testing but did not on Debian 8.
   https://bugzilla.gnome.org/show_bug.cgi?id=757254

So in the end I reworked both by putting the metadata in a regular 
GQueue and using the PTS timestamp to match elements in that queue to 
GStreamer's buffers.


Patches and changes from v10:
  server: Check the client video codec capabilities
      Fixed the case where the server and client cannot agree on the 
      video codec to use.
  server: Let the administrator pick the video encoder and codec
      The video_codecs GArray is now considered immutable which avoids 
      copying around and avoids having to make the
      RED_WORKER_MESSAGE_SET_VIDEO_CODECS RPC synchronous.
  server: Use the optimal number of threads for VP8 encoding
      Using the optimal number of cores for the VP8 encoder is now in a 
      separate patch (though skipping it will cause conflicts with 
      patches 14 and 16).
  server: Avoid copying the input frame in the GStreamer encoder
      Changed the zero-copy approach to not require ref/unref() of 
      RedDrawable objects to be thread-safe. Thread-safety aspects are 
      instead handled in gstreamer-encoder.c now.
  server: Adjust the frame rate based on the GStreamer encoding time
      This patch limits the frame rate (i.e. drops frames) when the 
      server has trouble keeping up with the encoding. This code seems 
      to only be needed in SpiceDeferredFPS mode (so maybe it's in the
      SpiceDeferredFPS code that something should be modified). In 
      any case this patch can be skipped without any impact on the rest 
      of the series.
  server: Respect the GStreamer encoder's valid bit rate range
      set_gstenc_bitrate() should now pass the right integer type to 
      g_object_set().
  server: Add support for GStreamer 0.10
      Reduced the number of #ifdefs for GStreamer 0.10 using 
      Christophe's suggestions.
  spice-gtk: Add a GStreamer video decoder for MJPEG, VP8 and h264
    - Tweaked some function names in the client's GStreamer decoder to 
      avoid potential name conflicts.
    - Fixed a race condition in the client which would cause it to 
      freeze when hovering madly over the seek barin YouTube videos.
    - As a side effect the new code now schedules the frames rending 
      itself so it can adapt when the mm-time get yanked around.
    - It also does not queue the frames in the pipeline and thus is less 
      likely to lose them when size changes force us to rebuild it.
  spice-gtk: Avoid GAsyncQueue for compatibility with GLib < 2.46
      This patch is optional and avoids the dependency on 
      g_async_queue_push_front() which is missing in GLib < 2.46 
      (e.g. for Debian 8).

-- 
Francois Gouget <fgouget at codeweavers.com>