[Spice-devel] question: how to test the gstreamer:h264 with qemu ?

[Frediano Ziglio]

> 
> On Thu, 4 Aug 2016, Frediano Ziglio wrote:
> [...]
> > In the Perfect World (TM) guest will send the video stream directly to
> > the client without server using much CPU. Detecting this would require to
> > have a driver in the client which support streaming (like VAAPI or DirectX
> > VA)
> > to get the byte stream from the guest. Possibly the guest could read back
> > the decoded video so in the real world most likely client AND spice-server
> > would have to do the decoding but this would avoid
> >   decoding (guest) -> encoding (server) -> decoding (client)
> > for a
> >   byte stream (guest) -> decoding (server) -> decoding (client)
> 

Obviously we are speaking "in theory".

> Implementing a passthrough streaming mechanism that works well would
> actually be pretty hard.
> 
> * First I can confirm that the most popular multimedia applications
>   (Firefox & Chrome's HTML5 video player, Flash, mplayer) use hardware
>   acceleration to decode the video but *not* to display it (if you look
>   for an exception there's Totem). So say you intercept the video stream
>   at the decoder stage and send it to the client. Where should the
>   client display it?
> 

The current h/w acceleration allows to allocate the buffer and decode
into that buffer. As you allocate that buffer you can easily understand
(a copy from that specific area/buffer) if client is using it.
Not saying that's easy it require some page toggling from Qemu and/or
the driver to support drawing from this buffer that is a much more aware
guest driver.
Yes, the application would have to use this buffer for drawing to the
screen in order to understand the relationship between original byte
stream and final screen at this point you know if is better to send the
original stream or not (because for instance you display only small part
of the video).

> * In fact the guest applications could very well not be displaying the
>   decoded video. That's the case when you transcode a video. In that
>   case streaming the video to the client would make no sense.
> 

In this case the buffer is not copied directly to screen.

> * But let's say the guest application does display the video. Now you
>   must prevent the spice server from sending the corresponding screen
>   updates either through streaming or regular updates. Otherwise you'll
>   end up doubling the bandwidth usage! (and wasting CPU cycles if it is
>   streaming them)
> 

Yes, you would have to just change the "output" of the streaming to the
destination buffer.

> * However the video detected by the server may not have the same size as
>   the original stream. This is almost always the case when you display a
>   video fullscreen for instance. Furthermore the spice server often
>   detects a video as two videos (e.g. one with the first 100 lines or
>   so, and another with the bottom 140, causing banding in YouTube for
>   instance). So it seems that matching the video being displayed with
>   the intercepted video stream would require solving that issue.
> 

Still this is possible to detect with a guest driver that support operation
like stretch (or 3D!).
About the split, yes, this should be fixed. But with 3D support for instance
the application usually display the whole frames.

> * But it's also possible that parts of the video are simply clipped.
>   Think scrolling a YouTube web page down until the video is only
>   partially visible. Or a dialog popping up in front of the video. Such
>   clipping must not only be detected on the Spice server when attempting
>   to match screen updates with the original video stream, it must also
>   be replicated when the client displays the frames it decoded.
>   Otherwise the video will obscure areas it's not supposed to (e.g. the
>   browser's title bar and tabs).
> 

Our protocol supports clipping. Still sending the clipping require a
more aware guest driver.

> * A variant is that a lot of applications apply overlays to the video,
>   some of them are even translucent: think YouTube ads/comments or movie
>   subtitles. Now you need to send those to the client so it can display
>   them on top the video it decoded. But how do you detect these
>   overlays? It seems like you'll have to decode the video on the server,
>   scale it just right, diff it with the video displayed by the guest
>   application, and send the diff to the client. All that efficiently
>   both from a server CPU usage point of view, and from a network
>   bandwidth one. (otherwise what's the point?)
> 

Yes, quite hard, would need to understand how application handle
the overlay, if using double buffer, drawing directly on top of
the stream or whatever. This could be quite overkilling :(
On the other side considering the possible bandwidth and CPU reduction
a bit of duplication with the overlay would not be a problem.

> * One way to avoid the Spice server-side issues would be for the decoder
>   to return fake content that's constant, like a green screen. Then the
>   Spice server no longer needs to match screen updates since there are
>   none. But you still have all the issues with transcoding applications,
>   figuring out where to display the video on the client, clipped videos,
>   overlay detection and handling, sending translucency information, etc.
> 
> * Then there's all the niggly details like expanded codec support, what
>   happens when a passthrough-capable client connects or disconnects
>   while a video is already playing, or what happens when a
>   passthrough-capable client connects at the same time as a
>   non-passthrough-capable one, etc.
> 

The possible guest driver would advise the codec supported and application
stop using our decoder if new profiles are not implemented.
Yes, supporting connection/disconnection would require to have control
of the internal compression state to send it at the beginning or simply
in this case you can reencode everything as we are doing now.

> 
> So unfortunately intercepting the original video stream is the easy
> part. The hard part is all the rest :-(
> 

Yes, we are not in the Perfect World (TM) :)

I think in the future we'll have to work more on the guest driver side.
And to support stream pass-throught that's far far much distance to go.

Frediano