[Spice-devel] [spice v14 00/29] Add GStreamer support for video streaming
Pavel Grunt
pgrunt at redhat.com
Wed May 18 07:40:21 UTC 2016
Hi Francois,
I pushed the spice-gtk part upstream.
Thanks!
Pavel
On Wed, 2016-05-04 at 11:40 +0200, Francois Gouget wrote:
> This patch series adds support for using GStreamer to encode and
> decode the video streams, adding support for VP8 and h264 codecs.
>
> As before the patches can also be grabbed from the gst branch of the
> repositories below:
>
> spice: https://github.com/fgouget/spice
> spice-gtk: https://github.com/fgouget/spice-gtk
> xf86-video-qxl: https://github.com/fgouget/xf86-video-qxl
> spice-protocol: https://github.com/fgouget/spice-protocol
>
> See also gst-sync for the old spice-gtk code. (there's also 'extras'
> branches with more experimental/future patches for the curious)
>
> Note also that I added gst-v10, gst-v11, gst-v12 and gst-v13 branches to
> help comparing these revisions.
>
> Note that to test GStreamer support with QEMU you probably need to grab
> the patch below and set the video-codecs option to something like
> gstreamer:h264 or gstreamer:vp8.
>
> https://lists.freedesktop.org/archives/spice-devel/2015-May/019771.html
>
>
> Changes from v13:
>
> Rebased + some changes suggested by Christophe Fergeau.
>
>
> Changes from v12:
>
> This round focused on performance and reliability in various special
> cases such as low bandwidth operation.
>
> * The server no longer rebuilds the pipeline from scratch every time
> the frame size or bitrate changes. The trick to it is to transition
> to the NULL state (instead of the PAUSED one as before).
> * I also unified configure_pipeline() and reconfigure_pipeline() by
> moving all the one-time configuration operations to
> create_pipeline() and tracking the configuration changes to apply.
> * The server can now also apply bitrate changes directly if the codec
> advertises it can be changed in the PLAYING state. This is the case
> for the x264enc codec.
> * The server can now detect when the GStreamer pipeline error occurs
> instead of getting stuck waiting for the compressed buffer.
> * I fixed the value for the error-resilient vp8enc option.
> * I removed the resize-allowed=true vp8enc setting as tests showed the
> decoder was sometimes not restoring a full-size frame.
> * I removed the 'frame' SpiceGstVideoBuffer field as it was not used.
> * push_raw_frame() now avoids allocating memory for chunk_copy() if
> zero_copy() was able to take care of the whole frame.
>
> * The client no longer rebuilds or even reconfigure the pipeline from
> scratch every time the frame size changes. It turns out that the
> decoders don't care (unlike the encoders).
> * I have protected the PTS calculation against underflows. This fixes
> freezes that could occasionally happen with negative latencies near
> the start of a stream (i.e. in low bandwidth cases).
> * I renamed SpiceFrameMeta to just SpiceFrame since it also contains
> the sample once the frame has been decoded, and is thus not just
> metadata. I also cleaned up the naming of the corresponding queues.
> * The GStreamer pipeline sometimes returns multiple buffers with the
> same PTS. The first buffer would remove its SpiceFrame from the
> decoding_queue so that the next one would not find a match, thus
> clearing the decoding_queue in the process. As a result the next
> buffer would not find a match either, again clearing the
> decoding_queue in the process, etc. So buffers that have an unknown
> PTS no longer clear the decoding_queue.
> * When frames are late for display the client now tries to display the
> least out of date one in the display_queue so there are at least
> some screen updates.
> * I removed the qos=true appsink option. It's not clear that it was
> helping so I decided to remove it until there is more evidence that
> it is needed.
> * I renamed stream_dropped_frame() to
> stream_dropped_frame_on_playback() to make it clearer that it's
> unrelated to the stream report data (which is based on arrival time).
>
>
> Besides rebases, the changes from v11 are limited to spice-gtk.
> Specifically they fix two compatibility issues:
> * v11 used a GAsyncQueue and relied on g_async_queue_push_front() which
> is only available in GLib >= 2.46 which is unfortunately not
> available in Debian 8 and presumably on many other Linux
> distributions.
> * It also associated the original network message with the
> corresponding GStreamer buffer using the GstMeta API. This is
> necessary in order to know where to display the decoded frame as this
> could change from one frame to the next. However some GStreamer
> elements have (refcounting) bugs that caused the metadata to be lost
> in the pipeline. I had a workaround in place that worked on Debian
> Testing but did not on Debian 8.
> https://bugzilla.gnome.org/show_bug.cgi?id=757254
>
> So in the end I reworked both by putting the metadata in a regular
> GQueue and using the PTS timestamp to match elements in that queue to
> GStreamer's buffers.
>
>
> Patches and changes from v10:
> server: Check the client video codec capabilities
> Fixed the case where the server and client cannot agree on the
> video codec to use.
> server: Let the administrator pick the video encoder and codec
> The video_codecs GArray is now considered immutable which avoids
> copying around and avoids having to make the
> RED_WORKER_MESSAGE_SET_VIDEO_CODECS RPC synchronous.
> server: Use the optimal number of threads for VP8 encoding
> Using the optimal number of cores for the VP8 encoder is now in a
> separate patch (though skipping it will cause conflicts with
> patches 14 and 16).
> server: Avoid copying the input frame in the GStreamer encoder
> Changed the zero-copy approach to not require ref/unref() of
> RedDrawable objects to be thread-safe. Thread-safety aspects are
> instead handled in gstreamer-encoder.c now.
> server: Adjust the frame rate based on the GStreamer encoding time
> This patch limits the frame rate (i.e. drops frames) when the
> server has trouble keeping up with the encoding. This code seems
> to only be needed in SpiceDeferredFPS mode (so maybe it's in the
> SpiceDeferredFPS code that something should be modified). In
> any case this patch can be skipped without any impact on the rest
> of the series.
> server: Respect the GStreamer encoder's valid bit rate range
> set_gstenc_bitrate() should now pass the right integer type to
> g_object_set().
> server: Add support for GStreamer 0.10
> Reduced the number of #ifdefs for GStreamer 0.10 using
> Christophe's suggestions.
> spice-gtk: Add a GStreamer video decoder for MJPEG, VP8 and h264
> - Tweaked some function names in the client's GStreamer decoder to
> avoid potential name conflicts.
> - Fixed a race condition in the client which would cause it to
> freeze when hovering madly over the seek barin YouTube videos.
> - As a side effect the new code now schedules the frames rending
> itself so it can adapt when the mm-time get yanked around.
> - It also does not queue the frames in the pipeline and thus is less
> likely to lose them when size changes force us to rebuild it.
> spice-gtk: Avoid GAsyncQueue for compatibility with GLib < 2.46
> This patch is optional and avoids the dependency on
> g_async_queue_push_front() which is missing in GLib < 2.46
> (e.g. for Debian 8).
>
More information about the Spice-devel
mailing list