[Spice-devel] question: how to test the gstreamer:h264 with qemu ?

[Francois Gouget]

On Thu, 4 Aug 2016, Frediano Ziglio wrote:
[...]
> In the Perfect World (TM) guest will send the video stream directly to
> the client without server using much CPU. Detecting this would require to
> have a driver in the client which support streaming (like VAAPI or DirectX VA)
> to get the byte stream from the guest. Possibly the guest could read back
> the decoded video so in the real world most likely client AND spice-server
> would have to do the decoding but this would avoid
>   decoding (guest) -> encoding (server) -> decoding (client)
> for a
>   byte stream (guest) -> decoding (server) -> decoding (client)

Implementing a passthrough streaming mechanism that works well would 
actually be pretty hard.

* First I can confirm that the most popular multimedia applications 
  (Firefox & Chrome's HTML5 video player, Flash, mplayer) use hardware 
  acceleration to decode the video but *not* to display it (if you look 
  for an exception there's Totem). So say you intercept the video stream 
  at the decoder stage and send it to the client. Where should the 
  client display it?

* In fact the guest applications could very well not be displaying the 
  decoded video. That's the case when you transcode a video. In that 
  case streaming the video to the client would make no sense.

* But let's say the guest application does display the video. Now you 
  must prevent the spice server from sending the corresponding screen 
  updates either through streaming or regular updates. Otherwise you'll 
  end up doubling the bandwidth usage! (and wasting CPU cycles if it is 
  streaming them)

* However the video detected by the server may not have the same size as 
  the original stream. This is almost always the case when you display a 
  video fullscreen for instance. Furthermore the spice server often 
  detects a video as two videos (e.g. one with the first 100 lines or 
  so, and another with the bottom 140, causing banding in YouTube for 
  instance). So it seems that matching the video being displayed with 
  the intercepted video stream would require solving that issue.

* But it's also possible that parts of the video are simply clipped. 
  Think scrolling a YouTube web page down until the video is only 
  partially visible. Or a dialog popping up in front of the video. Such 
  clipping must not only be detected on the Spice server when attempting 
  to match screen updates with the original video stream, it must also 
  be replicated when the client displays the frames it decoded. 
  Otherwise the video will obscure areas it's not supposed to (e.g. the 
  browser's title bar and tabs).

* A variant is that a lot of applications apply overlays to the video, 
  some of them are even translucent: think YouTube ads/comments or movie 
  subtitles. Now you need to send those to the client so it can display 
  them on top the video it decoded. But how do you detect these 
  overlays? It seems like you'll have to decode the video on the server, 
  scale it just right, diff it with the video displayed by the guest 
  application, and send the diff to the client. All that efficiently 
  both from a server CPU usage point of view, and from a network 
  bandwidth one. (otherwise what's the point?)

* One way to avoid the Spice server-side issues would be for the decoder 
  to return fake content that's constant, like a green screen. Then the 
  Spice server no longer needs to match screen updates since there are 
  none. But you still have all the issues with transcoding applications, 
  figuring out where to display the video on the client, clipped videos, 
  overlay detection and handling, sending translucency information, etc.

* Then there's all the niggly details like expanded codec support, what 
  happens when a passthrough-capable client connects or disconnects 
  while a video is already playing, or what happens when a 
  passthrough-capable client connects at the same time as a 
  non-passthrough-capable one, etc.


So unfortunately intercepting the original video stream is the easy 
part. The hard part is all the rest :-(


-- 
Francois Gouget <fgouget at codeweavers.com>