[Spice-devel] video passthrough ideas

[Zeeshan Ali (Khattak)]

Hi everyone,
   As some of you already know, I've been looking into the needed
"video passthrough" feature: i-e sending the encoded stream to client
and decode it on client-side. Although I'm still investigating on how
to best handle this, I thought I already share my thoughts/findings so
far so if I'm going in the wrong directions, team members can
correct/guide me. Here is the two options I've been looking into:

1. VAAPI (Video Acceleration API): The idea here is to implement a
custom VAAPI driver for the guest that redirects all decoding/playback
calls to the client through a spice channel. On the client side, we
create a channel handler based on gstreamer and its playbin2 element.

Issues/challenges here are:

  a. We still need to figure out how our VAAPI driver running in
user-space (its loaded as shared-object in the application) will
communicate to the spice server running inside Qemu?

  b. GStreamer currently do no have working VAAPI plugins and AFAIK,
it won't have any for at least a few months more.

  c. Portability: As the name suggests, VAAPI is a very linux-specific AFAICT.

  d. Limited/fixed number of encodings supported: the profiles are
defined as enums and AFAICT you can't magically add profiles on the
fly.

  e. Haven't yet figured how the interaction with QXL/windowing-system
on client can/will work (i-e telling QXL to leave the part of the
screen where the video is being played on as blank etc), need to read
more of QXL and VAAPI code to figure i guess.

2. GStreamer all the way: This is the idea that I originally came-up
with when Christophe suggested a "magic GStreamer element" on IRC. The
idea here is pretty similar to the VAAPI one except that instead of
writing a VAAP driver, we write a gstreamer sink element that would
run on the guest. This sink element will send the gstreamer data
through spice-server to the client and the story on the client is then
very similar. Here we can make use of GStreamer Data Protocol (GDP)
for which there already exists elements to serialize/deserialize the
stream into/from. Here is the advantages that comes to mind of this
approach:

  a. We can make use of the dynamic nature of GStreamer and be able to
handle any media type for which the client has plugins for.

  b. GStreamer has been known to work on windows (and even Mac) so
this option SHOULD be more easily portable.

Issues/challenges here are:

  a. Same as issue 1.a above.

  b. GDP is unidirectional so we'll need to figure a way to serialize
upstream events from the client. Shouldn't be difficult though once we
know how to solve the previous issue.

  c. Same as 1.e above.

-- 
Regards,

Zeeshan Ali (Khattak)
FSF member#5124