GStreamer API over RPC: does it make sense?

Wed Nov 6 09:11:23 PST 2013

On Tue, Nov 5, 2013 at 10:59 PM, Marc-André Lureau <mlureau at redhat.com> wrote:
>
>
> ----- Original Message -----
>> Hi Sebastian,
>>
>> Thanks for your comments! Very encouraging for me.
>> Are you talking about SoC 2014? Where can I look at the status of this
>> project? Is it awaiting approval from Google now?
>
> I would also be interested to look at the proposal :)
>
>> As for what we really need for Spice - I'm not sure I understand your
>> idea correctly. There are 3 major media-related use cases we want to
>> optimize:
>> * playback of a local media resource (local - at VM); this is
>> addressed at http://www.spice-space.org/page/Features/CodecPassthrough
>> * playback of a remote media resource
>> * IP telephony
>> Last two are addressed at
>> http://www.spice-space.org/page/Features/MediaRedirection
>>
>> Your idea may be relevant for CodecPassthrough feature - when a file
>> is already at VM and we don't want to transcode it, but rather send it
>> directly to the client over Spice channel. Implementing custom VAAPI
>> driver was considered as a possible approach, but later dropped (I'm
>> wasn't around at that time, not aware of the reasons actually). But
>
> I am not sure why you say it was dropped. This is complicated, talking is always cheap ;) the fact is nobody really spent time on implementing any of this, it's just an idea that was discussed from time to time. Feel free to give it a try. To me it looks like it's quite limited in codec support, and might be harder to implement than a few GStreamer sinks.
>

I've come to this idea because all mentions of VAAPI have been removed
from the feature's wiki page... BTW I agree with your points.

>> again, we want to pass-through the media data without decoding it - so
>> we need to intercept it earlier - before it is decoded by e.g.
>> GStreamer and delivered to PulseAudio and/or rendered to virtual video
>> device.
>>
>> The essence of my MediaRedirection feature is media stream totally
>> by-passing VDI server/VM. This means incoming and outgoing media
>> streams must be handled by Spice client (or its extension of some
>> kind). So I don't understand how this can be done at lower level...
>>
>> On the other hand, the GStreamer API isn't ideal because:
>> * Not all applications are using GStreamer; there are other engines like VLC
>> * Spice supports both Linux and Windows guests - and GStreamer isn't
>> that popular in the latter environment.
>
> But directshow is, perhaps it's possible to implement a directshow filter that would use gstreamer? But I don't have enough background on Windows development, also those things tend to change after every release. Something like EVR? (http://msdn.microsoft.com/en-us/library/ms694916.aspx) DXVA would be the equivalent for VAAPI, I think.
>
Agree. Moreover, AFAIK something like this is already implemented by
Citr*x, and exactly with using GStreamer at the client side.

>
> Tbh, given the complexity of all this, I would start with a solution that works for gstreamer Linux guest, and later try to extend (or invent something else) for other platforms.
>
> And eventually, having a gstreamer pass-through and a lower level dxva/vaapi wont be incompatible.
>

Seems like a good plan to me. I'm going to try this approach and see
how it goes...

>> So I'm open for better suggestions - how to off-load media processing
>> from a VM to the client in a universal way.
>>
>> I'm also thinking about something like "Media Control" protocol or
>> extension to Spice protocol - and it can be implemented to support
>> multiple media engines via wrappers of some kind... Direct binary RPC
>> of proto-buffers just seems much more simpler and robust...yet fairly
>> mechanical solution.
>>
>>
>> On Tue, Nov 5, 2013 at 1:06 AM, Sebastian Dröge
>> <sebastian at centricular.com> wrote:
>> > On Mo, 2013-11-04 at 23:39 +0400, Fedor Lyakhov wrote:
>> >> Hello GStreamer developers,
>> >>
>> >> I'm working on a Media Redirection feature
>> >> (http://www.spice-space.org/page/Features/MediaRedirection) for Spice,
>> >> a system to access remote virtual machines (similar to VNC or RDP, but
>> >> better).
>> >>
>> >> Essentially, we want to optimize following use cases in VDI:
>> >> * Playback of remote media resource, i.e. a media stream from remote
>> >> server (e.g. user watches Youtube)
>> >> * Telecommunications, i.e. IP telephony
>> >>
>> >> The idea is to have e.g. media player at virtual machine (at VDI
>> >> server) to link against a stub library instead of normal GStreamer.
>> >> This library is capable to send API calls to real GStreamer library
>> >> located at user's PC (client) - the actual RPC implementation can be
>> >> done via Google Protocol Buffers for instance. The transport of RPC
>> >> calls will be done within dedicated Spice channel (or e.g. over simple
>> >> TCP connection, to begin with).
>> >>
>> >> My question is - does this scheme make any sense to you? Is it viable?
>> >> I'd appreciate any comments!
>> >>
>> >> Maybe it has been considered by someone before?
>> >> What potential issues we'd need to overcome? Any blockers?
>> >> Maybe we can do this for part of API, sufficient to run common media
>> >> applications? We're concerned with following application types mostly:
>> >> media player, web browser (e.g. Firefox using GStreamer) and IP
>> >> softphones (e.g. Empathy).
>> >
>> > It's one of the things I proposed as a project for summer of code this
>> > year :) I think it's feasible to do with the GStreamer API, but you
>> > might run into unexpected problems when actually trying this.
>> >
>> > That said I don't think abstracting the GStreamer API is really what you
>> > want for Spice. To me it sounds more like you would like to have
>> > something more low-level that just allows to output/input raw or
>> > compressed audio/video and send that over the Spice channel. You're
>> > probably only interested in direct data flow, not a complete abstraction
>> > of the GStreamer API. Something like a remote PulseAudio server plus a
>> > bit more ;)
>> >
>> > --
>> > Sebastian Dröge <sebastian at centricular.com>
>> > Centricular Ltd - http://www.centricular.com
>> > Expertise, Straight from the Source
>> >
>> > _______________________________________________
>> > gstreamer-devel mailing list
>> > gstreamer-devel at lists.freedesktop.org
>> > http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>> >
>>
>>
>>
>> --
>> Best regards,
>> Fedor
>>

-- 
Best regards,
Fedor