How does a muxer deal with packet loss?

[Michiel Konstapel]

Somewhat hypothetical, but it may be what I am currently running into: how
do muxers deal with missing data in their inputs?

Consider a pair of unreliable network sources, like RTP from udpsrc, or in
my case a webrtcbin, with an audio and video stream. Due to poor network
conditions, most or all audio gets through, but lost video packets mean
some key frames don't make it through and subsequent delta frames can't be
decoded, so a few seconds of video never makes it into the pipeline, but
there is audio.

After some processing and encoding, the audio and video go into a muxer
(mpegtsmux in my case).

- How does the muxer deal with the time periods in which there are audio
buffers, but no video buffers? Does it freeze, waiting for video timestamps
that will never come? Does it output only audio and no video in that time?

- I *assume* it doesn't just freeze, but how does it know when to stop
waiting - maybe when a video buffer with a higher PTS arrives? Does it then
send out all the audio in between?

- Is it wise to always put a leaky queue in front of muxers to ensure all
pipeline branches can maintain flow? I fear that my pipeline is freezing
because my video branch stalls because a demuxer can't output buffers,
because the audio branch is blocked at the muxer downstream, because no
correspondencing video is available, deadlocking the system. Can that
happen, and if so, how to prevent it? If not, good - but why not, what
prevents it?

Hoping for some insight from people who understand some of the internals
and fundamentals better than I do :)
Cheers,
Michiel

-- 
Michiel Konstapel

*Lead Software Developer*
*aanmelder.nl <http://aanmelder.nl>*

T: +31 (0)15 2400119
E: michiel at aanmelder.nl
[image: aanmelder.nl] <https://www.aanmelder.nl/i/footer>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210620/8d04d1a9/attachment.htm>