Syncing of audio and video coming from different pipelines

[Stuart Marshall]

Hi, 

I am sending captured h264 video from one pipeline and captured audio (aac) from a second pipeline, both coming off the same machine using gstreamer 1.8.3. The two streams are being send by different invocations of gst-launch-1.0.

I am trying to receive the streams in a single pipeline and mux them into an mp4 file. That single pipeline is also on gstreamer 1.8.3.

I’m having trouble getting the audio and video to by in sync. I watched Sebastian’s presentation at the 2015 gstreamer conference. My understanding is that the two sending pipelines will use different RTP sequences, but will send RTCP packets referencing the same NTP reference clock (the machine’s NTP time). On the receiving pipeline I have set rtpbin ntp-sync to true. My understanding is that this will get the receiving pipeline’s rtpjitterbuffer to use the NTP times in the RTCP packets to correctly sync up the two streams. However, the resulting mp4 often has the video and audio out of sync, with the audio delayed by 0 to 2 seconds (approximately).

One suspicion I had was that the NTP sync wouldn’t work until an RTCP packet had arrived, so I’ve been testing the audio video sync after the recording has been going for a while (10 seconds and more). But no luck from that.

Any suggestions on how I can fix the sync issue?

I suspect one option is to send the audio and video from the same pipeline, but I’d like to avoid doing that. My app sends multiple video streams and a single audio stream, and I wish to be able to turn the video streams on and off independently. There’s probably a way to do this while having them in the same pipeline, but it seems like there ought to be a solution with the NTP times in the RTCP packets.

Thanks,

Stuart