Audio synchronization between individual RTP audio streams

Tue Nov 9 16:08:47 UTC 2021

Hi Tim,

Thank you for the response.
We found some relevant RFCs that may help with our problem, as you
mentioned. I guess we're talking about the RFC 6051 and the
so-called in-band synchronization mechanism.
We found the issue that was opened about 4 years ago:
https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/-/issues/383
I guess it wasn't merged to any branch or version of Gstreamer yet. Is this
the correct issue thread and patch? Are we on the right way?

___
Thank you,

*Vitaliy Lazarev*
Software Engineer,
notAnotherOne <http://notanotherone.com/>

E-mail: vlazarev at notanotherone.com
T: +7 911 296-73-93

On Fri, Nov 5, 2021 at 4:33 PM Tim Müller via gstreamer-devel <
gstreamer-devel at lists.freedesktop.org> wrote:

> Hi Vitaliy,
>
> > We have a system with one RX pipeline that is running on RPi4 and
> > listening to several UDP sources with different ports, and near 10
> > individual TXs. All of the devices are located in the same room. TXs
> > are sending their own RTP stream over Wi-Fi to RX simultaneously +
> > RTCP with SR and SDES. And after some time due to network connection
> > issues and other factors, these TX audio streams are running out of
> > audio sync (from 100ms to 1s audio lag).
> > We found a similar issue, and it says that if there is no RTCP
> > connection between RX and TXs, there will be no synchronization. But
> > this issue is about hosts that are not located in the same physical
> > space.
> > Is RTCP usage can help to prevent audio lag between individual RTP
> > streams (maybe feedback with RRs)? Can it be solved only using RX
> > side with some manipulations with pipeline elements (rtpbin NTP/RTCP
> > sync options), RTP timestamps, or something?
> > Is this even possible to get a several milliseconds synchronization
> > between many RTP audio streams, that are located physically in one
> > room?
>
> The general "problem" with RTP is that the timestamps in the RTP
> packets are offset randomly without any absolute reference or base. A
> receiver will typically just record the timestamp on the first packet
> and map that to some local 0 base time and then interpolate from that.
>
> If a sender sends audio and video or multiple audio streams, that might
> mean that the receiving streams could initially be out of sync.
>
> Sender report (SR) RTCP packets provide extra information from the
> sender to receivers. They contain mappings between RTP timestamps to an
> "ntp time". This then provides a receiver with a common time base for
> all streams coming from a sender, so it can use that to offset
> audio/video streams accordingly and achieve lipsync. A receiver doesn't
> necessarily know what these ntp timestamps refer to though, so it can
> only use it to sync the incoming streams relatively, but not to map it
> to an absolute or local time or clock.
>
> Now if you have multiple senders there's another problem: Different
> machines will be using different clocks, and clocks drift over time.
>
> So what you want to do is ideally make all devices (or at least all
> senders) use a common clock. This can be an NTP clock or a PTP clock or
> a GstNetClock tracking a local GstNetTimeProvider.
>
> Once you've done that you make the senders use that clock for sender
> report ntp time (for bonus points configure sender to use capture time
> instead of send time for SRs, but depends a bit on your hardware and
> drivers how well that will work).
>
> Then all senders will basically be using the same clock/time as
> reference for their ntp time stamps, and the receiver can correlate the
> streams from the different senders.
>
> There are also RFCs for RTP header extensions that allow senders to put
> ntp timestamps into each packet they send out which allows for rapid
> synchronisation (no need to wait for a Sender Report), but you still
> need for senders to agree on a clock/time reference in order fo this to
> be useful.
>
> I believe if you configure the AVPF rtp profile it will send a SR
> immediately at the beginning instead of only sending it after a few
> seconds.
>
> Good luck!
> Cheers
>  Tim
>
>

-- 
___

© 2021, NotAnotherOne Corp. Address: 8 The Green, Suite #4466, Dover, 
DE, 19901, USA, Technological singularity Company Limited, 197198, Russian 
Federation, Saint-Petersburg, Krasnogo Kursanta str. 25, lit. Zh, bld. 1, 
floor 7, suite 307. This email may contain confidential information 
protected by legal privilege. NOTANOTHERONE CORP and/or Technological 
singularity Company Limited is the proprietor of that confidential 
information. Any review, disclosure, copying, distribution and use without 
consent from proprietor are prohibited. If you are not the intended 
recipient, please notify us and delete this email and its attachments from 
your operation system and disk drive.

© 2021, ООО «Технологическая 
сингулярность», 197198, Российская Федерация, г.  Санкт-Петербург, ул. 
Красного Курсанта, д. 25, лит. Ж, этаж 7 (семь), пом. 307, NotAnotherOne 
Corp. Адрес: 8 The Green, Suite #4466, Dover, DE, 19901, USA. Данное 
электронное сообщение может содержать информацию ограниченного доступа, 
обладателем которой является Общество с ограниченной ответственностью 
«Технологическая сингулярность» и/или NOTANOTHERONE CORP. ЗАПРЕЩАЕТСЯ 
осуществлять любое ознакомление, раскрытие, разглашение, копирование, 
распространение такой информации без специального согласия ООО 
«Технологическая сингулярность» и/или NOTANOTHERONE CORP. Если Вы не 
являетесь адресатом сообщения или не имеете обязательств по защите 
конфиденциальности такой информации, пожалуйста, незамедлительно сообщите 
об этом нам и удалите настоящее письмо со всеми приложениями.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20211109/f1cb1a8a/attachment-0001.htm>