Debugging audio/video synchronization

[Marianna Smidth Buschle]

How can I debug audio/video synchronization issues?


Basically I have a long pipeline that I decoupled in many pieces using 
Ridge Run interpipesrc/sink.

In some situations I end up having problems with the synchronization of 
audio and video, more specifically the video seems to be late in 
relation to the audio.

However I don't know how to debug that, what to look for to explain and 
fix the problem...

I have previously ran into problems using the interpipesrc/sink because 
of segment handling. So I kind of expect something similar.


The basic idea is:


- Having N camera sources:

v4l2src ! h264_enc ! mpegtsmux ! rtppay ! rtpbin

udpsrc ! rtpbin ! rtpdepay ! tsdemux ! h264parse ! interpipesink name=cam1_V


- And a common audio source which gets mixed for each camera

audiosrc ! deinterlace name=d. ! interpipesink name=audio_ch0 d. ! 
interpipesink name=audio_ch1

interleave name=i ! interpipesink name=cam1_A \

audiomixer name=left ! i. \

audiomixer name=right ! i. \

interpipesrc listen-to=audio_ch0 ! left. interpipesrc 
listen-to=audio_ch1 ! left. \

interpipesrc listen-to=audio_ch0 ! right. interpipesrc 
listen-to=audio_ch1 ! right.


- Then an "input-selector" (running on H264) to decide on which camera 
to show:

interpipesrc listen-to=cam1_V name=V_select ! h264_dec ! intepipesink 
name=live_V

interpipesrc listen-to=cam1_A name=A_select ! intepipesink name=live_A


- Another "input-selector" (running on NV12) to be able to do replay 
(seeking from file):

interpipesrc listen-to=live_V name=V_replay_select ! intepipesink 
name=video_out

interpipesrc listen-to=live_A name=A_replay_select ! intepipesink 
name=audio_out


- Result gets muxed as MPEG-TS and streamed out

mpegtsmux name=m ! rtppay ! udpsink \

interpipesrc listen-to=audio_out ! audioconvert ! aac_enc ! m. \

interpipesrc listen-to=video_out ! h264_enc ! h264parse ! m. \


The interpipesrc allows for choosing the time of stream synchronization: 
which is basically either to keep the current timestamps or to 
re-timestamp the buffers.

My audio/video synchronization problem occurs when I setup 
V_replay_select and A_replay_select to re-timestamp the buffers (which I 
have to do if I use replay, in order to match the timestamps from the 
file with the live stream).

If I keep the original timestamps then audio and video are in synch, but 
then my replay won't work.

Note that I re-timestamp the buffers in V_select and A_select (which 
operate on H264) and there it isn't a problem. It only seems to be a 
problem when I do it in the RAW stream (NV12).


Any what can be wrong or how can I debug this?

-- 
Best regards / Med venlig hilsen
“Marianna Smidth Buschle”