Synchronizing separate audio and video streams based on rtp timestamp

[Wedge, Ryan J. (JSC-CD4)[SGT, INC]]

We've encountered a non-standard synchronization issue in a new system. A platform running GStreamer will receive two multicast streams, one for audio and another for video. The audio stream will only be present while there is actual intelligible audio. A concern is that the audio may be encoded with much less delay than the video and received by our platform unsynchronized, such that the audio will be out of sync with the video when output to speakers/monitor.

I'm hoping someone with more knowledge of GStreamer's limitations and capabilities can provide input on whether this is a trivial problem that can be solved using an existing GStreamer 1.x deployment. A proxy-like application prior to GStreamer's udpsrc could be implemented to perform this synchronization upstream of GStreamer entirely.

This proxy-like application may do the following:
Receive video packets, extract current RTP timestamp. Receive initial audio packet, extract RTP timestamp. Compare Audio RTP timestamp to current Video RTP timestamp. If Video Timestamp >= Audio Timestamp: send audio packet immediately. Else, buffer audio packets with spacing information until Video RTP timestamp >= Audio RTP timestamp. Then, empty the buffered audio at the rate it was received. Assumes video is always delayed more than audio.


Ryan Wedge
Software Engineer, MCCS Communications Subsystem


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20190826/1feb52ca/attachment.html>