Issue with decklinksrc - Precaution for any live src element

[Tim-Philipp Müller]

On Wed, 2013-07-03 at 10:26 -0700, Baby Octopus wrote:

> I'm using decklinksrc element to capture video from an SDI camera and
> streaming it over the network after encoding
> 
> *gst-launch-1.0 decklinksrc mode=10 name=src src.! video/x-raw ! tee ! queue
> ! h264enc ! mpegtsmux name=mux ! udpsink host=239.1.1.12 port=5050 src.!
> audio/x-raw ! tee ! queue ! audioresample ! lamemp3enc ! mux.*
> 
> Here are the issues that I saw, which is probably not taken care in
> decklinksrc though its a live src
> 
> 1. The pipeline hangs whenever h264enc fails to run realtime(for above
> 720p). How should a source skip frames if the encoder can't run realtime?
> 2. When I add videoscale and feed 360p to h264enc, I see ocassionally a
> particular frame still having part of older frame. This older frame is
> always 60frames behind the current frame. I see src writing into the buffer
> when h264enc is reading it. I see some buffer lock missing or release not
> happening properly
> 
> As I see, this is a trivial issue a live src can encoder. Can somebody throw
> some light on how to *skip frames* and *release buffer properly*
> 
> I have reproduced these issues consistently. I would like to file bug. If
> somebody can throw some light, I would like to fix and give patch

udpsink could (does already?) send QoS messages when rendering packets.
This would let upstream elements (encoder, source) know whether the
encoder is too slow (its output too late). Then either the encoder can
react to that and pick some cheaper/faster algorithms, or it can send it
upstream to the source, which could drop frames if the long-term average
indicates that the encoder not being able to keep up is more than just a
short-term glitch.

You can also set the queue to be leaky, then frames will get dropped if
the queue overflows.

The source writing into a buffer that the encoder is processing sounds
like a bug.

Cheers
 -Tim