How does a muxer deal with packet loss?

[Tim Müller]

Hi Michiel,

> Somewhat hypothetical, but it may be what I am currently running
> into: how do muxers deal with missing data in their inputs?

Depends on the muxer/format and also the GStreamer version you're
using, because a muxer may have been ported to the new GstAggregator
base class in a more recent version of GStreamer.

 
> Consider a pair of unreliable network sources, like RTP from udpsrc,
> or in my case a webrtcbin, with an audio and video stream. Due to
> poor network conditions, most or all audio gets through, but lost
> video packets mean some key frames don't make it through and
> subsequent delta frames can't be decoded, so a few seconds of video
> never makes it into the pipeline, but there is audio.
> 
> After some processing and encoding, the audio and video go into a
> muxer (mpegtsmux in my case). 
> 
> - How does the muxer deal with the time periods in which there are
> audio buffers, but no video buffers? Does it freeze, waiting for
> video timestamps that will never come? Does it output only audio and
> no video in that time? 

It depends. If the muxer has been ported to GstAggregator, there's a
chance it can handle live inputs (i.e. this case) properly, that is
continue to output data even if one of the inputs has dropped out. In
that case it will operate in "live" mode and will timeout and process
the data it has if it doesn't have data on all pads within the
configured latency.

However, I think that will only work if it has already received some
input data (or rather: has received a CAPS event telling it the format
of the input data), so the moving-on-without-data-on-all-pads might not
work if there's data missing at the very beginning.

(I haven't tried it or double-checked the code, this is just from
memory.)

> - I *assume* it doesn't just freeze, but how does it know when to
> stop waiting - maybe when a video buffer with a higher PTS arrives?
> Does it then send out all the audio in between?

In live mode it will stop waiting after a while (configured upstream
latency + muxer latency).

In non-live mode (e.g. file inputs, appsrc in non-live mode) it will
need either data on all pads or a GAP event on pads where there's no
data to make it tick along.


> - Is it wise to always put a leaky queue in front of muxers to ensure
> all pipeline branches can maintain flow?

A leaky queue is more of an "emergency valve" to make sure that an
overflow (slow processing in one pipeline branch / downstream part of a
pipeline) doesn't affect processing in the data producing upstream part
of a pipeline, e.g. you might configure a leaky queue after splitting
the pipeline with a tee element to make sure that if that branch isn't
processing/consuming data quickly enough, that the branch pushing into
the tee won't get blocked on that queue running full (thus starving any
other branches that might feed off that same tee).


>  I fear that my pipeline is freezing because my video branch stalls
> because a demuxer can't output buffers, because the audio branch is
> blocked at the muxer downstream, because no correspondencing video is
> available, deadlocking the system. Can that happen, and if so, how to
> prevent it? If not, good - but why not, what prevents it?

Depends. I'm not sure I fully follow. Ideally one would want to make
sure the muxer doesn't block, but it maybe can't be avoided initially
when it's waiting for the input caps event.

You can put a leaky queue into the branches feeding into the muxer if
you want to make sure the muxer not processing data doesn't affect
those upstream receiving/depayloading branches (esp. if you tee those
off e.g. for display).

It might be possible to do something better here too.

Do you re-encode or save the audio/video as they come in over RTP?

Cheers
 Tim