GStreamer API over RPC: does it make sense?

[Fedor Lyakhov]

Hi Sebastian,

Thanks for your comments! Very encouraging for me.
Are you talking about SoC 2014? Where can I look at the status of this
project? Is it awaiting approval from Google now?

As for what we really need for Spice - I'm not sure I understand your
idea correctly. There are 3 major media-related use cases we want to
optimize:
* playback of a local media resource (local - at VM); this is
addressed at http://www.spice-space.org/page/Features/CodecPassthrough
* playback of a remote media resource
* IP telephony
Last two are addressed at
http://www.spice-space.org/page/Features/MediaRedirection

Your idea may be relevant for CodecPassthrough feature - when a file
is already at VM and we don't want to transcode it, but rather send it
directly to the client over Spice channel. Implementing custom VAAPI
driver was considered as a possible approach, but later dropped (I'm
wasn't around at that time, not aware of the reasons actually). But
again, we want to pass-through the media data without decoding it - so
we need to intercept it earlier - before it is decoded by e.g.
GStreamer and delivered to PulseAudio and/or rendered to virtual video
device.

The essence of my MediaRedirection feature is media stream totally
by-passing VDI server/VM. This means incoming and outgoing media
streams must be handled by Spice client (or its extension of some
kind). So I don't understand how this can be done at lower level...

On the other hand, the GStreamer API isn't ideal because:
* Not all applications are using GStreamer; there are other engines like VLC
* Spice supports both Linux and Windows guests - and GStreamer isn't
that popular in the latter environment.

So I'm open for better suggestions - how to off-load media processing
from a VM to the client in a universal way.

I'm also thinking about something like "Media Control" protocol or
extension to Spice protocol - and it can be implemented to support
multiple media engines via wrappers of some kind... Direct binary RPC
of proto-buffers just seems much more simpler and robust...yet fairly
mechanical solution.


On Tue, Nov 5, 2013 at 1:06 AM, Sebastian Dröge
<sebastian at centricular.com> wrote:
> On Mo, 2013-11-04 at 23:39 +0400, Fedor Lyakhov wrote:
>> Hello GStreamer developers,
>>
>> I'm working on a Media Redirection feature
>> (http://www.spice-space.org/page/Features/MediaRedirection) for Spice,
>> a system to access remote virtual machines (similar to VNC or RDP, but
>> better).
>>
>> Essentially, we want to optimize following use cases in VDI:
>> * Playback of remote media resource, i.e. a media stream from remote
>> server (e.g. user watches Youtube)
>> * Telecommunications, i.e. IP telephony
>>
>> The idea is to have e.g. media player at virtual machine (at VDI
>> server) to link against a stub library instead of normal GStreamer.
>> This library is capable to send API calls to real GStreamer library
>> located at user's PC (client) - the actual RPC implementation can be
>> done via Google Protocol Buffers for instance. The transport of RPC
>> calls will be done within dedicated Spice channel (or e.g. over simple
>> TCP connection, to begin with).
>>
>> My question is - does this scheme make any sense to you? Is it viable?
>> I'd appreciate any comments!
>>
>> Maybe it has been considered by someone before?
>> What potential issues we'd need to overcome? Any blockers?
>> Maybe we can do this for part of API, sufficient to run common media
>> applications? We're concerned with following application types mostly:
>> media player, web browser (e.g. Firefox using GStreamer) and IP
>> softphones (e.g. Empathy).
>
> It's one of the things I proposed as a project for summer of code this
> year :) I think it's feasible to do with the GStreamer API, but you
> might run into unexpected problems when actually trying this.
>
> That said I don't think abstracting the GStreamer API is really what you
> want for Spice. To me it sounds more like you would like to have
> something more low-level that just allows to output/input raw or
> compressed audio/video and send that over the Spice channel. You're
> probably only interested in direct data flow, not a complete abstraction
> of the GStreamer API. Something like a remote PulseAudio server plus a
> bit more ;)
>
> --
> Sebastian Dröge <sebastian at centricular.com>
> Centricular Ltd - http://www.centricular.com
> Expertise, Straight from the Source
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>



-- 
Best regards,
Fedor