[Spice-devel] [spice v14 00/29] Add GStreamer support for video streaming

[Pavel Grunt]

Hi Francois,

I pushed the spice-gtk part upstream.

Thanks!
Pavel

On Wed, 2016-05-04 at 11:40 +0200, Francois Gouget wrote:
> This patch series adds support for using GStreamer to encode and 
> decode the video streams, adding support for VP8 and h264 codecs.
> 
> As before the patches can also be grabbed from the gst branch of the
> repositories below:
> 
> spice:          https://github.com/fgouget/spice
> spice-gtk:      https://github.com/fgouget/spice-gtk
> xf86-video-qxl: https://github.com/fgouget/xf86-video-qxl
> spice-protocol: https://github.com/fgouget/spice-protocol
> 
> See also gst-sync for the old spice-gtk code. (there's also 'extras'
> branches with more experimental/future patches for the curious)
> 
> Note also that I added gst-v10, gst-v11, gst-v12 and gst-v13 branches to 
> help comparing these revisions.
> 
> Note that to test GStreamer support with QEMU you probably need to grab 
> the patch below and set the video-codecs option to something like 
> gstreamer:h264 or gstreamer:vp8.
> 
> https://lists.freedesktop.org/archives/spice-devel/2015-May/019771.html
> 
> 
> Changes from v13:
> 
> Rebased + some changes suggested by Christophe Fergeau.
> 
> 
> Changes from v12:
> 
> This round focused on performance and reliability in various special 
> cases such as low bandwidth operation.
> 
>  * The server no longer rebuilds the pipeline from scratch every time 
>    the frame size or bitrate changes. The trick to it is to transition 
>    to the NULL state (instead of the PAUSED one as before).
>  * I also unified configure_pipeline() and reconfigure_pipeline() by 
>    moving all the one-time configuration operations to 
>    create_pipeline() and tracking the configuration changes to apply.
>  * The server can now also apply bitrate changes directly if the codec 
>    advertises it can be changed in the PLAYING state. This is the case 
>    for the x264enc codec.
>  * The server can now detect when the GStreamer pipeline error occurs 
>    instead of getting stuck waiting for the compressed buffer.
>  * I fixed the value for the error-resilient vp8enc option.
>  * I removed the resize-allowed=true vp8enc setting as tests showed the 
>    decoder was sometimes not restoring a full-size frame.
>  * I removed the 'frame' SpiceGstVideoBuffer field as it was not used.
>  * push_raw_frame() now avoids allocating memory for chunk_copy() if 
>    zero_copy() was able to take care of the whole frame.
> 
>  * The client no longer rebuilds or even reconfigure the pipeline from 
>    scratch every time the frame size changes. It turns out that the 
>    decoders don't care (unlike the encoders).
>  * I have protected the PTS calculation against underflows. This fixes 
>    freezes that could occasionally happen with negative latencies near 
>    the start of a stream (i.e. in low bandwidth cases).
>  * I renamed SpiceFrameMeta to just SpiceFrame since it also contains 
>    the sample once the frame has been decoded, and is thus not just 
>    metadata. I also cleaned up the naming of the corresponding queues.
>  * The GStreamer pipeline sometimes returns multiple buffers with the 
>    same PTS. The first buffer would remove its SpiceFrame from the 
>    decoding_queue so that the next one would not find a match, thus 
>    clearing the decoding_queue in the process. As a result the next 
>    buffer would not find a match either, again clearing the 
>    decoding_queue in the process, etc. So buffers that have an unknown 
>    PTS no longer clear the decoding_queue.
>  * When frames are late for display the client now tries to display the 
>    least out of date one in the display_queue so there are at least 
>    some screen updates.
>  * I removed the qos=true appsink option. It's not clear that it was 
>    helping so I decided to remove it until there is more evidence that 
>    it is needed.
>  * I renamed stream_dropped_frame() to 
>    stream_dropped_frame_on_playback() to make it clearer that it's 
>    unrelated to the stream report data (which is based on arrival time).
> 
> 
> Besides rebases, the changes from v11 are limited to spice-gtk. 
> Specifically they fix two compatibility issues:
>  * v11 used a GAsyncQueue and relied on g_async_queue_push_front() which 
>    is only available in GLib >= 2.46 which is unfortunately not 
>    available in Debian 8 and presumably on many other Linux 
>    distributions.
>  * It also associated the original network message with the 
>    corresponding GStreamer buffer using the GstMeta API. This is 
>    necessary in order to know where to display the decoded frame as this 
>    could change from one frame to the next. However some GStreamer 
>    elements have (refcounting) bugs that caused the metadata to be lost 
>    in the pipeline. I had a workaround in place that worked on Debian 
>    Testing but did not on Debian 8.
>    https://bugzilla.gnome.org/show_bug.cgi?id=757254
> 
> So in the end I reworked both by putting the metadata in a regular 
> GQueue and using the PTS timestamp to match elements in that queue to 
> GStreamer's buffers.
> 
> 
> Patches and changes from v10:
>   server: Check the client video codec capabilities
>       Fixed the case where the server and client cannot agree on the 
>       video codec to use.
>   server: Let the administrator pick the video encoder and codec
>       The video_codecs GArray is now considered immutable which avoids 
>       copying around and avoids having to make the
>       RED_WORKER_MESSAGE_SET_VIDEO_CODECS RPC synchronous.
>   server: Use the optimal number of threads for VP8 encoding
>       Using the optimal number of cores for the VP8 encoder is now in a 
>       separate patch (though skipping it will cause conflicts with 
>       patches 14 and 16).
>   server: Avoid copying the input frame in the GStreamer encoder
>       Changed the zero-copy approach to not require ref/unref() of 
>       RedDrawable objects to be thread-safe. Thread-safety aspects are 
>       instead handled in gstreamer-encoder.c now.
>   server: Adjust the frame rate based on the GStreamer encoding time
>       This patch limits the frame rate (i.e. drops frames) when the 
>       server has trouble keeping up with the encoding. This code seems 
>       to only be needed in SpiceDeferredFPS mode (so maybe it's in the
>       SpiceDeferredFPS code that something should be modified). In 
>       any case this patch can be skipped without any impact on the rest 
>       of the series.
>   server: Respect the GStreamer encoder's valid bit rate range
>       set_gstenc_bitrate() should now pass the right integer type to 
>       g_object_set().
>   server: Add support for GStreamer 0.10
>       Reduced the number of #ifdefs for GStreamer 0.10 using 
>       Christophe's suggestions.
>   spice-gtk: Add a GStreamer video decoder for MJPEG, VP8 and h264
>     - Tweaked some function names in the client's GStreamer decoder to 
>       avoid potential name conflicts.
>     - Fixed a race condition in the client which would cause it to 
>       freeze when hovering madly over the seek barin YouTube videos.
>     - As a side effect the new code now schedules the frames rending 
>       itself so it can adapt when the mm-time get yanked around.
>     - It also does not queue the frames in the pipeline and thus is less 
>       likely to lose them when size changes force us to rebuild it.
>   spice-gtk: Avoid GAsyncQueue for compatibility with GLib < 2.46
>       This patch is optional and avoids the dependency on 
>       g_async_queue_push_front() which is missing in GLib < 2.46 
>       (e.g. for Debian 8).
>