Audiomixer dropping input when mixing live and non-live sources

[Nicolas Dufresne]

Le dimanche 12 mars 2017 à 13:41 -0500, Sean DuBois a écrit :
> Hi Nicolas, thanks for the quick response
> 
> > Can you clarify how it fails for you? You need some latency for
> > this to
> > work, but 2s might bee to much. You would need enough latency on
> > rtp jitter
> > buffer too.
> 
> The audiomixer drops all incoming buffers from my rtmpsrc, the output
> is completely
> silent (but if I add a audiotestsrc it works, proving that the
> audiomixer is working)

Interesting, that looks like everything is being dropped, as it's late.
 I'm wondering if rtmpsrc is really implementing a live source. A live
source would set timestamp base on arrival time, while this problem
seems to indicate that it sets timestamp from zero or something,
regardless of how much time it took to start. What you could try
though, is to pause you pipeline first, wait a bit, and then start it.

> 
> How would I know the magnitude of latency I should be adding? It
> seems
> variable, 2 seconds works and sometimes it doesn't. Is there a signal
> from the audiomixer pad that I could watch and up the latency until
> it
> works?

rtmp is TCP, so yes, it's variable, un-controlled. Ideally, the
rtmpsource element would compute the initial latency, and advertise it
over the pipeline should it could all work. Then just a small amount in
the mixer for jitter would be sufficient.

> 
> The other issue is that when I do add latency and RTP sources the
> compositor starts to act very poorly. This makes sense to me, I don't
> want any
> latency from my RTP input if it doesn't arrive in time I am ok
> discarding it. However for my RTMP I know that the video/audio is
> fine,
> it just seems to have 'fallen behind' because a PadProbe that adds 2
> seconds to it fixes everything. I just don't know what I should be
> measuring to figure out where those 2 seconds are coming from.

The mixing latency is to compensate small delays between inputs, 0 is a
little racy.

> 
> 
> Here is my RTP input, however even if I do get latency+RTP working it
> still doesn't change the issue that I don't know how much latency to
> add.
> ```
> gst-launch-1.0 udpsrc port=5000 caps="application/x-rtp" !
> rtph264depay ! decodebin ! videoconvert ! compositor
> latency=5000000000 sink_1::xpos=500 name=c ! autovideosink
> videotestsrc is-live=true ! c.

You are missing an rtpjitterbuffer. That means your packet could endup
in the wrong order, time information will be jittery too. This is far
from ideal when doing RTP. It will also make the compositor start
really early, which then make the other streams appear late.

> 
> gst-launch-1.0 videotestsrc ! x264enc speed-preset=veryfast !
> rtph264pay ! udpsink port=5000 host="127.0.0.1"
> ```
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20170312/c89b4058/attachment.sig>