Synchronizing separate audio and video streams based on rtp timestamp

[Wedge, Ryan J. (JSC-CD4)[SGT, INC]]

Thanks so much for your response Tim. 

> Usually the sender would also send Sender Report (SR) RTCP packets to
> receivers, and these packets will map the RTP timestamps for each
> stream to a common NTP timebase. GStreamer's rtpbin will automatically
> take care of synchronising the streams in a GStreamer context in that
> case.
Having the source send RTCP SR packets may not be possible, so we are looking for a solution independent of RTCP. Does rtpbin rely on SR packets for synchronization? I did not see a fallback method mentioned in the plugin docs.  

> If you don't have RTCP packets, there are multiple RTP header
> extensions that will carry a common reference time and that can then be
> used for the purpose of synchronising streams.
We will investigate what is open to change on the sender. The extension flag is not currently set on the video RTP headers. Browsing through some of the RTP packets in wireshark shows the timestamp field actually jumping around quite dramatically. Sometimes single units, sometimes 10k units, others 50k units within milliseconds of wallclock time. I assume this is related to i-frames and p-frames and the 90KHz clock. 

> Or the sender can make an SDP available with a mapping.
I researched the possible parameters within an SDP and didn't see a field that appeared to describe a value used to sync the streams. Can you please elaborate on how the SDP can help with synchronization? 

> Do you have control over the sender as well? Is it GStreamer?
We do not have control over either the audio or video sender and neither originate from the same physical device. I am not aware of whether the senders are using GStreamer. The two sending devices will have their system clocks synchronized.

You have absolutely shown there exist several standard solutions to this type of multi-stream synchronization issue, but from what is available for us to change from the receiver only side, it's not looking like we have many standard options open to us. I'll continue researching rtpbin, RTP extensions, RTSP, and RTCP packets and try to get a better understanding of where we lie in the solution space without requesting sender changes.  

Thanks,
Ryan