Inconsistent/variable delay between USB and onboard alsasrc

[Michiel Konstapel]

On 21-07-2021 19:17, Nicolas Dufresne via gstreamer-devel wrote:
> Le mercredi 21 juillet 2021 à 16:58 +0200, Michiel Konstapel via gstreamer-devel
> a écrit :
>> I am recording audio from two alsasrc elements: an analog line input
>> from the onboard sound card, and a USB audio device (USB microphone). I
>> need the recordings to be synchronized, so playing them back together
>> doesn't produce a nasty "tiled bathroom" sound. I expected the different
>> sources to have different internal delays, which I can adjust for with a
>> probe that adjusts the buffer timestamp.
>>
>> However, there seems to be no consistency in the time difference: in one
>> run, the analog input is 3 ms behind the USB audio, the next time it's
>> 44 ms, then it's 20 ms... I measured this by placing the microphones
>> side by side, producing a clap or tick and comparing the tracks in Audacity.
>>
>> I've tried
>> - always choosing the onboard card as the clock source
>> - choosing neither and using the system clock
>> - all the options for alsasrc slave-method except "none"
>>
>> How should I set things up so the sources are reliably synchronized?
> 
> This one is not as much about slaving as it is about getting HW timestamp. First
> step, make sure your driver supports that. Run your pipeline with GST_DEBUG=4
> (info level), and looks for that trace:
> 
> https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/blob/master/ext/alsa/gstalsasrc.c#L276
> 
> Then monitor for:
> https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/blob/master/ext/alsa/gstalsasrc.c#L993
> 
> This will first ensure that you have timestamp that aligns more closely to
> reality.

That first line only prints if the alsasrc is not the pipeline clock, 
correct? Specifically, the pipeline clock must be GST_TYPE_SYSTEM_CLOCK.

If I set the alsa sources to provide-clock=false then it prints "Using 
driver timestamps !" for both. However, I never see the second line 
("ALSA timestamp"). I don't see any of the warnings/errors 
gst_alsasrc_get_timestamp can print, either.

> Then the synchronization does not depends on the src, but on the sink
> element. By default, this is set to slave-method "skew" with alignment-threshold
> of 40ms. Meanwhile even if you have non-drifting audio, it will simply break the
> stream to fix it once the delay falls passed this threshold. That makes your
> observation very consistent, since with a margin of error, your result ranges
> within 40ms, without drift, it will stay at that position.
> 
> skew is pretty much the only working option in alsasink atm, play with
> alignment-threshold and drift-tolerance in order to get something closer to 4ms.

We don't have an alsasink in the pipeline; the audio and video get muxed 
into mpegts and sent over the network (SRT). Does that mean it's solely 
up to the sources to get their synchronization right?

> An alternative is to mix the streams inside gstreamer, but that introduce a lot
> of latency. audiomixer will do a better job at aligning the streams.

That's interesting. How much is "a lot" of latency? We already have ~700 
ms of latency on the network transmission, so adding tens of 
milliseconds is not a problem.

We need the different microphones as separate channels, so we can't mix 
them into one channel. However, I just realized audiomixer outputs 
multiple channels. If we feed it, for example, the two microphones as 
the left and right channel, will it do the aligning between them, but 
keep them as separate channels in the output?

Cheers,
Michiel