pitch element breaks lip sync in Chrome

[Nicolas Dufresne]

Sorry for top posting, but this seems like a good bug report to me. Can
you paste this on a bug in bugs.gnome.org please ?

Le jeudi 06 septembre 2018 à 19:11 +0200, Juan Navarro a écrit :
> Hi,
> 
> It seems that the 'pitch' filter (in the soundtouch plugin, 
> gst-plugins-bad) introduces some kind of wrong timestamp, clock skew, 
> bad sequence, a combination of those, or maybe something different (but 
> probably related).
> 
> I'm seeing an accumulative delay in Chrome between video and audio in a 
> WebRTC call (_not_ using the new GStreamer's WebRTC element, yet) where 
> the source is sending a video+audio stream, and the audio is filtered 
> with the pitch element. Chrome is unable to perform the lip sync 
> successfully, and for some reason deduces that somehow the audio is 
> lagging behind the video (which is actually not), so it delays the video 
> indefinitely until the delay gets to Chrome's maximum, 10 seconds.
> 
> The net effect of this issue is practically the same as what happened in 
> this Chrome bug: 
> https://bugs.chromium.org/p/webrtc/issues/detail?id=5456 (just check the 
> screenshots)
> with 'googCurrentDelayMs' and 'goodMinPlayoutDelayMs' growing linearly. 
> At that time it happened to be Chrome wrongly using the webcam's 
> timestamp, which had a different clock rate than the system's timestamp.
> 
> But, in this case I don't think it's a Chrome bug; I have verified that 
> this is caused by the GStreamer's 'pitch' filter, by sourcing this 
> simple test pipeline to my custom WebRTC source element:
> 
> ... -> (raw audio)
>    -> audioconvert -> audioresample -> pitch ->
>    -> audioconvert -> audioresample -> WebRTC
> 
> (Probably most, if not all of those audioconvert/audioresample elements 
> are not needed, I added them just to fall on the safe side)
> 
> This generates the mentioned delay in the video presentation handled by 
> Chrome.
> 
> However nothing of this happens if the 'pitch' element is removed and 
> any other is used, e.g. an 'scaletempo' element:
> 
> ... -> (raw audio)
>    -> audioconvert -> audioresample -> scaletempo ->
>    -> audioconvert -> audioresample -> WebRTC
> 
> This produces a normal lip sync result in Chrome. Delay (latency) stays 
> at around 100, 150 ms.
> 
> I've been reading google's WebRTC code, wanting to know exactly what is 
> the name of the value that is to blame:
> https://cs.chromium.org/chromium/src/third_party/webrtc/video/stream_synchronization.cc
> 
> but finding out what is the correct function chain is difficult, and I'm 
> still not sure of exactly *what* is making Chrome confused and wrongly 
> assuming that the audio is behind the video, when it's not.
> 
> I have cherry-picked and applied all commits that touched the file 
> './ext/soundtouch/gstpitch.cc' into a custom built version of 
> gst-plugins-bad, but the issue persists so it's not a matter of trying 
> the latest code (after a lot of time without changes, the pitch filter 
> received some patches in June so I wanted to test if those helped...)
> 
> Only idea I have is that the pitch element is missing some sequence 
> number handling, or something about the pipeline's clock rate... but I'm 
> out of ideas.
> 
> 
> Please help :)
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: This is a digitally signed message part
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20180906/29623529/attachment.sig>