Choppy audio playing a videofile through webrtcsink

[Philipp B]

Hi,

[dislaimer: I didnt work with gstreamer for some time, my terminology
might be a bit rusty]

I also had audio issues when streaming video files over webrtc, and
judging by your video, it could be the same issue you face.

For me it turned out to be small rounding errors in the audio frame
timestamps. Due to the realtime nature, browsers playing webrtc audio
are not aiming for a perfect smooth playback, but rather for perfect
realtime. Small deviations in timestamps instantly lead to the browser
inserting small pieces of silence, or dropping a few ms of audio.

The most obvious effect was the volume being way lower than it should
be. This was caused by the player/browser constantly fading segments
in and out, to overlap them with the next one, or with silence. Also,
I had similar distortion artifacts than you have.

To be clear, my issue seemed to be related to transmission of
non-live, pre-encoded video only. WebRTC is made for live
transmissions, where audio frame timestamps are directly related to
the wall clock time at which a segment has been recorded. In contrast,
a non-live video file typically has a stream of audio segments, which
are expected to be played back without gaps, even when there are small
gaps according to the time stamps.

What happens if you increase the audio frame size? In my case, this
was improving the situation dramatically, but not fully fixing it.

In the end, I got this solved by "smoothing" audio time stamps. I
wrote my own gstreamer element, that kept an estimation of the next
frames timestamp, based on the last frame. In case, there is a small
deviation (e.g. below 5 ms), the frames timestamp will be changed to
that of the estimation. I can provide the code of that element in case
your interested.

Philipp

Am Mi., 15. Nov. 2023 um 23:39 Uhr schrieb Vincent Deconinck via
gstreamer-devel <gstreamer-devel at lists.freedesktop.org>:
>
> Hi,
>
> I made some progress in my quest to stream video files to a browser using webrtcssink, but audio is not OK. I built the following pipeline:
> +-----------------------------------------------------------------------------+
> | uridecodebin.audio ->- audioconvert ->- audioresample ->- webrtcssink.audio |
> | uridecodebin.video ----->----- videoconvert ------>------ webrtcssink.video |
> +-----------------------------------------------------------------------------+
>
> The webrtcsink pads are statically linked, and the uridecodebin pads are dynamically linked upon "pad_added" signal (as in https://gstreamer.freedesktop.org/documentation/tutorials/basic/dynamic-pipelines.html ).
>
> Problem: The video looks OK in the browser, but the audio is choppy. A quick and dirty (sorry) video tells a thousand words: https://youtu.be/1EbPHLZvVLc?t=27
>
> (using signalling server, JS API and gstreamer built 2 days ago from git HEAD)
>
> I was first using the http URI to the source video but I also tried using a local copy of the file, to no avail.
>
> Any idea what could be wrong ? Or an idea how I could track down that issue ?
>
> Kind regards,
>
> Vincent