appsink and autovideosink frames

[jreilly]

On 64-bit Windows 7, using gst 1.8.2, I am writing an application that does
some frame processing on an RTSP stream. My pipeline looks like this (the
URIs are omitted for brevity):

   rtspsrc location=aaa latency=0 name=bbb ! rtph264depay ! h264parse ! tee
name=t1
      t1. ! queue ! mp4mux ! filesink location="ccc"
      t1. ! queue ! decodebin ! appsink max-buffers=1 drop=true
name=myAppSink
      t1. ! queue ! decodebin ! videoconvert ! cairooverlay name=myCairo !
autovideosink name=myVideo

I set the appsink callbacks, and enable the cairo draw signal. The video is
being hosted inside a window in a wxWidgets 3.0 application. When I run, I
get my video in the file, appsink_new_sample() gets called, and the cairo
draw signal handler gets called. If my appsink just produces some canned
results (rectangles), they draw smoothly on the video.

When the appsink_new_sample() frame processing is *very* minimal, everything
works great. When the frame processing takes a long time (100-200 ms),
things go to heck. At that time, the video is very framey and freezes often.
It will freeze for seconds, and then jump by many seconds. The video file
looks great.

(I have tried queue2 and multiqueue with the same results)

The surprise is that the appsink_new_sample() function gets called often,
and 10-100 times more often than the cairo draw.

Given that every pipeline "segment" is in a queue and thus a thread, I would
expect the video to be independent of the appsink. But the appsink seems to
process frame-by-frame and the video seems starved. I also get video sink
bus messages that say something like "very many frames are being dropped."
That is obvious.

My goal is for the annotated (i.e. draw a few rectangles) video to be smooth
(even 5 Hz is fine) and tolerate varying processing times in the appsink. Do
I need to copy the frame buffer in the appsink_new_sample() and then hand
that copy off to a new thread on my own?

Any ideas will be appreciated.

Thanks.

-reilly.




--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/appsink-and-autovideosink-frames-tp4679461.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.