rtpbin: ts adjustment needs filtering/smoothing

Sat Oct 17 20:23:39 UTC 2020

Nicolas,

Sorry for my tone, if that offended. I Thought your response was a little snarky and maybe that get me riled up. I am a bit frustrated after spending 4 weeks working on improving the synchronization only to start listening to the audio and hearing the ticks/pops. But let’s move on.

Can you point me to the “50 lines of code” you mention – I believe you are referring to the resampler code? Is that in GstAudioSink? I would like to look at it.

Also, the post/thread that I mentioned was sent to me via email from the gstreamer-devel list on freedesktop.org. The author is Pavel Hofman and the title is “Adaptive resampling in gstaudiobasesink.c”. It was posted last WED, 10/14. I will pop this up again by posting a reply to the message. Pavel is interested in working on the adaptive resampling, so if there is a better way for him to bring up the topic and how to write a patch please let him/me know. Thanks

-Charlie

From: gstreamer-devel <gstreamer-devel-bounces at lists.freedesktop.org> On Behalf Of Nicolas Dufresne
Sent: Saturday, October 17, 2020 1:13 PM
To: Discussion of the development of and with GStreamer <gstreamer-devel at lists.freedesktop.org>
Subject: Re: rtpbin: ts adjustment needs filtering/smoothing

Le sam. 17 oct. 2020 13 h 00, <charleslaub at sbcglobal.net <mailto:charleslaub at sbcglobal.net> > a écrit :

>Perhaps you didn't notice the default slave-method, which is skew. Skew will remove or fill with silence the audio stream to fix the synchronization >between the audio card clock and the NTP clock. As it does that naively, you will ear ticks and pop. You'll also notice the resample method, which in >theory should work, but in practice the resampler there is buggy and the adjustment too aggressive. Some work is needed there to make it work.

As you say "buggy" is an understatement - it is TERRIBLE. It's essentially unlistenable because the "over aggressive" action introduces a warble to the music, like you are listening underwater.  

So, the only solution that comes to mind is custom slaving, which is not available to the general public? Hmmmm.

Would be nice to tone down a little. The code is open source, the hard work of plumbing this is in place, but no one felt like contributing a fix for this (resampling is expensive, so mostly for desktop, were audio deamon already do a much better job). And we are aware it never worked.

As for the custom slaving mode, it's publicly available, but you need audio HW with a controllable PLL, and the rounding + smoothing in such case will always be HW specific.

Perhaps in place of the buggy ALSA resampler, you could implement asynchronous resampling? This was 

This is pure C, there is no ALSA there. It is about 50 line of code. We have another resampler now, but no one looked it up to see if it can be used more dynamicly then how it is used in audioresample element.

brought up in a post a couple of days ago in this list under the subject "Adaptive resampling in gstaudiobasesink.c". I think it would be very worthwhile and would really improve the audio performance of Gstreamer in general.

Could you or someone follow up on that topic? 

I'm sorry, I don't remember this thread. Do you have links ? Perhaps a merge request would get more attention.

-Charlie

-----Original Message-----
From: gstreamer-devel <gstreamer-devel-bounces at lists.freedesktop.org <mailto:gstreamer-devel-bounces at lists.freedesktop.org> > On Behalf Of Nicolas Dufresne
Sent: Saturday, October 17, 2020 10:50 AM
To: Discussion of the development of and with GStreamer <gstreamer-devel at lists.freedesktop.org <mailto:gstreamer-devel at lists.freedesktop.org> >
Subject: Re: rtpbin: ts adjustment needs filtering/smoothing

Perhaps you didn't notice the default slave-method, which is skew. Skew will remove or fill with silence the audio stream to fix the synchronization between the audio card clock and the NTP clock. As it does that naively, you will ear ticks and pop. You'll also notice the resample method, which in theory should work, but in practice the resampler there is buggy and the adjustment too aggressive. Some work is needed there to make it work.

Well known companies using GStreamer have introduced the "custom"
slave-method. Along with
gst_audio_base_sink_set_custom_slaving_callback(), you get notified per units of the drift. These vendor controlls the HW, so instead of resampling, they will adjust the audio clock PLL in order to lock the audio to the system/ntp/ptp/etc. clock in use.
>  
> I would be happy to provide my pipelines if that would be helpful, 
> provide measurement data, or any other info that might lead to a 
> solution.
>  
> -Charlie
>  
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org <mailto:gstreamer-devel at lists.freedesktop.org> 
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
gstreamer-devel at lists.freedesktop.org <mailto:gstreamer-devel at lists.freedesktop.org> 
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
gstreamer-devel at lists.freedesktop.org <mailto:gstreamer-devel at lists.freedesktop.org> 
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20201017/757fe7bd/attachment.htm>