[gst-devel] Live Video Stream Mixer

[Florent]

Hi,

> This is quite a project. Not sure if it fits a final year project. I am not a
> video expert, but answer as no one else did so far.

I agree it is a large scope ; i tried something similar in the past.
What i found out is that synchronization and mixing from network
sources is really a challenge with Gstreamer (according to my
experience), especially with videomixer etc...

>> 1) Is live mixing of video streams going to be possible using the
>> GStreamer framework and are there any suggestions on how I should look
>> into implementing this? Can I simply cut between pipeline frames once
>> the video has been decoded?

It depends on what you will choose; if you need 'videomixer', then
each of your input streams will have to be decoded and keyed before
mixing (e.g. using 'alpha'); you could also try the same with
'switcher', although i never tried to use it. Also, audio will have to
be downmixed by using 'adder' plus 'volume' on input streams for
example...

If you can cope with raw DV, you could try switching the DV packets in
real time in gstreamer (like [1]). Seems like an easier project to me.
This way, you could have a simple pipeline, which would also be very
light in processing. However, you will not be able to have
swipes/openGL effects etc... (which would need decoding, mixing and
reencoding in any case).

>> 3) Low latency is desirable, how powerful a PC would be required for
>> this decoding, mixing and recoding multiple low-res live video streams

How many streams ? Which resolution and framerate, origin codec ?
Which target codec ? These questions only make sense with numbers, but
expect the need for powerful machines. As for latency, it will depend
of the number of filters and operations you will have between the
senders and the encoder.

>> and is there any graphics hardware acceleration available with GStreamer
>> using CUDA, OpenCL or similar?

Keep in mind that gstreamer is heavily multi-threaded (and able),
which is already a nice feature performance-wise. There currently are
some efforts to integrate deblocking and debayering in opengl in
gst-plugins-gl for Elphel cameras with GLSL, you have some
hardware-accelerated colorspace conversion in some plugins
(cluttergst, glupload)... And nothing prevents you to try coding the
first OpenCL gstreamer plugin. In other terms, there are alternatives
for accelerated processing.

> Besides all the gfx accelerated
> video stuff is quite new on linux, as hardware vendors are not really supportive
> here.

Most of the hardware graphics vendors are accelerating stream decoding
and display (vdpau/...), there are currently no linux-supported
encoding features (AFAIK). Some input hardware can be found on [2].

Cheers

Florent

[1] http://dvswitch.alioth.debian.org/wiki/
[2] http://www.mythtv.org/wiki/Category:Hardware