Appsrc/Appsink framerate issues

[Martin Goldstone]

Hi there,

I'm fairly new to gstreamer, so I'm probably missing something obvious
with this, but here goes:  I'm currently working on a project which
takes multiple sources, displays them in a ximagesink, and also puts a
couple of them through a videomixer, which will then be streamed out
over the network (in the future, at the moment I'm just throwing it at
another ximagesink).  The idea is that a user will be able to flick
between inputs, select a transition, etc.

As I want the ability to independently seek the sources, and would
also like a timeoverlay on the ximagesink of each source which will
reflect the runtime position of that source, I've reached the
conclusion that I have to deal with multiple pipelines, so now I have
a set up like this (I'm just using a videotestsrc at the moment)

Source pipeline: videotestsrc -> tee -> queue -> ffmpegcolorspace ->
timeoverlay -> ximagesink
Output pipeline: videotestsrc -> videomixer -> ffmpegcolorspace ->
timeoverlay ->ximagesink

After the user clicks a button to send the source through to the
videomixer (running in a separate pipeline), I end up with this:

Source pipeline: videotestsrc -> tee -> queue -> ffmpegcolorspace ->
timeoverlay -> ximagesink
                                                    -> queue -> appsink

Output pipeline: videotestsrc -> videomixer -> ffmpegcolorspace ->
timeoverlay ->ximagesink
 appsrc -> ffmpegcolorspace -> videomixer

I'm getting output through the output pipeline, and output also
continues through the source pipeline, but on the output pipeline, I
seem to be dropping some frames (I'm running at 25fps, setting
timestamps on the buffers in the function I have in between the
appsink and appsrc).  This seems to vary, as sometimes it appears to
be only a couple of frames a second (ie barely noticeable), but
sometimes I seem to be losing about half of the frames.  If I set sync
to false on the appsink, then my output pipeline plays perfectly
smoothly, at the correct rate, but my source pipeline's ximagesink
seems to be missing out half the frames.  If I set sync to false on
that, I get superfast output there, and normal speed output on the
output pipeline, then my appsink's buffer grows to a ridiculous size
and my system starts killing processes.

My next thought is to dispense with the tee, and just have the appsink
send the buffers to 2 appsrc's, but before I do that, does anyone have
any thoughts on what I might be doing wrong?  Have I gone completely
mad with this project?

Thanks,

Martin


PS For that extra element of craziness, I'm doing this with python.