[Spice-devel] MediaRedirection

[Francois Gouget]

On Mon, 12 Dec 2016, Kenny Zhang wrote:

> hi,
> 
> Recently I used spice as remote viewer with Qemu, it's powerful.
> Compared with freerdp, the video player, only support media player, it's
> limited, and I get yours MediaRedirection(
> https://www.spice-space.org/page/Features/MediaRedirection), it's cost much
> CPU usage, seems is too heavy. So I want to base your design. and change a
> little, to develop another one, here is my mind.
> 1. Guest OS have a media server that communicate with spice server.
> 2. Guest OS can support and developed video player communicate with media
> server(player send demux data and its position & size to spice client via
> spice server, and other control command).
> 3. Spice client decode data and playing.

I developped a MediaRedirection proof of concept for CodeWeavers. We 
call it a passthrough mode since it results in the video stream 
bypassing the Spice server entirely.

It works by leveraging Spice client's support for a wide range of video 
formats thanks to the GStreamer support, and GStreamer support in the 
guest. Here's how it works:

* You install a special GStreamer video sink in the guest, 
  spicevideosink, and give it a high priority so GStreamer will try to 
  use it rather than the standard video sinks.

* You also install a version of vdagentd with passthrough support.

* The spicevideosink then detects whether vdagentd and the current Spice 
  client support video passthrough. If so it accepts compressed video 
  and sends it to vdagentd which then routes it to the client using the 
  standard VDAgent protocol. It also sends data about the location and 
  size of the videosink window in the same way.

* The client handles the new video passthrough VDAgent messages and 
  creates the appropriate video decoder instance to display the video in 
  the right location.

* vdagentd is there to deal with X session switching: if a video plays 
  in an X session but the user switches to another, vdagentd tells the 
  client that this video stream is no longer visible and thus should no 
  longer be displayed. Conversely if the user switches back to the 
  session where the video is playing, vdagentd tells the client that 
  this stream is visible again.

* To the above one must add another video sink, smartvideosink, which is 
  there to overcome the limitations of autovideosink but is independent 
  of the actual passthrough. It is used so Totem picks the right video 
  sink.


As a proof of concept our current implementation has a number of 
limitations, some of which would require significant work to overcome.

* This only works with applications that use a GStreamer video sink 
  such as Totem. While other applications like Firefox use GStreamer to 
  decode the video, they then grab the decoded frames and display them 
  themselves (after applying overlays for instance). Since no GStreamer 
  video sink is involved this means spicevideosink cannot do anything to 
  handle this case. Overcoming this limitation is pretty tricky: 
  https://lists.freedesktop.org/archives/spice-devel/2016-August/031140.html

* The spicevideosink does not know which parts of its window are 
  visible. This means on the client the video gets displayed on top of 
  all other windows. That's can be ok in specific scenarios but it's 
  clearly not for general use.

* Switching from a client that supports passthrough to one that does 
  not while a video is playing, would require a complete reconfiguration 
  of the GStreamer pipeline on the guest: replace the video sink, figure 
  out how to decode the video, and likely add some element to convert 
  the raw video from YUV to RGB. I don't think that's supported 
  midstream.

* Conversely, switching from a client with no passthrough support 
  to one with passthrough support will not cause the video stream to 
  switch to passthrough since the GStreamer pipeline will have no reason 
  to change video sinks in the first place. Then you have cases of 
  multiple simultaneous clients with some supporting passthrough and 
  some not.

* Besides the above limitations the current code is not in a suitable 
  state for upstream inclusion. As a start spicevideosink is a GStreamer 
  0.10 video sink so it works on RHEL 6 :-(


-- 
Francois Gouget <fgouget at codeweavers.com>