Choppy audio playing a videofile through webrtcsink

[Philipp B]

Ive put the code here:
https://github.com/honkphluxx/gst-smoothts

Dont expect too much, this is just a hack for now.
You need to have a fixed audio frame size which you need to know at
the time the pipe is build.
(That can definetely be improved, it was just good enough to fit my purpose)

I use it, after encoding, like this:

audioconvert ! audioresample !
audio/x-raw,channels=2,clock-rate=48000 !
opusenc bitrate=128000 frame-size=60 !
smoothts ts-step=60000000 allowed-error=2000000 !
recodequeue.sink_{} recodequeue.src_{} !

(frame size 60ms, max smoothing interval is 2ms, both given in nanosec
to smoothts)

Most of the code in that repo is just what the gstreamer code
generator produced. Actual code is 8 lines here:
https://github.com/honkphluxx/gst-smoothts/blob/main/plugins/gstsmoothts.c#L274
 (and ofc feel free to use and license these lines as you like)



I never played with youtube. Logs from gstreamer didnt help in my
case, I needed to look into Chrome WebRTC stats to identify the
problem.
In case you need to go deeper, it might really help to reproduce in a
setup where you stream to your browser - Browsers have great tooling
to debug WebRTC quality issues.


Am Do., 16. Nov. 2023 um 17:52 Uhr schrieb cfd new <newcfd at yahoo.com>:
>
> Hi, Philipp,
>
>    very interested in your code. I got the following message when I stream a live video to youtube. Maybe it is related to your case.
>
>    Joe
>
> 0:03:35.140304346 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.039937500
> 0:03:35.141001930 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.039937500
> 0:03:35.141314636 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.039937500
>
>
> On Thursday, November 16, 2023, 06:35:55 a.m. EST, Philipp B via gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
>
>
> Hi,
>
> [dislaimer: I didnt work with gstreamer for some time, my terminology
> might be a bit rusty]
>
> I also had audio issues when streaming video files over webrtc, and
> judging by your video, it could be the same issue you face.
>
> For me it turned out to be small rounding errors in the audio frame
> timestamps. Due to the realtime nature, browsers playing webrtc audio
> are not aiming for a perfect smooth playback, but rather for perfect
> realtime. Small deviations in timestamps instantly lead to the browser
> inserting small pieces of silence, or dropping a few ms of audio.
>
> The most obvious effect was the volume being way lower than it should
> be. This was caused by the player/browser constantly fading segments
> in and out, to overlap them with the next one, or with silence. Also,
> I had similar distortion artifacts than you have.
>
> To be clear, my issue seemed to be related to transmission of
> non-live, pre-encoded video only. WebRTC is made for live
> transmissions, where audio frame timestamps are directly related to
> the wall clock time at which a segment has been recorded. In contrast,
> a non-live video file typically has a stream of audio segments, which
> are expected to be played back without gaps, even when there are small
> gaps according to the time stamps.
>
> What happens if you increase the audio frame size? In my case, this
> was improving the situation dramatically, but not fully fixing it.
>
> In the end, I got this solved by "smoothing" audio time stamps. I
> wrote my own gstreamer element, that kept an estimation of the next
> frames timestamp, based on the last frame. In case, there is a small
> deviation (e.g. below 5 ms), the frames timestamp will be changed to
> that of the estimation. I can provide the code of that element in case
> your interested.
>
> Philipp
>
> Am Mi., 15. Nov. 2023 um 23:39 Uhr schrieb Vincent Deconinck via
> gstreamer-devel <gstreamer-devel at lists.freedesktop.org>:
>
> >
> > Hi,
> >
> > I made some progress in my quest to stream video files to a browser using webrtcssink, but audio is not OK. I built the following pipeline:
> > +-----------------------------------------------------------------------------+
> > | uridecodebin.audio ->- audioconvert ->- audioresample ->- webrtcssink.audio |
> > | uridecodebin.video ----->----- videoconvert ------>------ webrtcssink.video |
> > +-----------------------------------------------------------------------------+
> >
> > The webrtcsink pads are statically linked, and the uridecodebin pads are dynamically linked upon "pad_added" signal (as in https://gstreamer.freedesktop.org/documentation/tutorials/basic/dynamic-pipelines.html ).
> >
> > Problem: The video looks OK in the browser, but the audio is choppy. A quick and dirty (sorry) video tells a thousand words: https://youtu.be/1EbPHLZvVLc?t=27
> >
> > (using signalling server, JS API and gstreamer built 2 days ago from git HEAD)
> >
> > I was first using the http URI to the source video but I also tried using a local copy of the file, to no avail.
> >
> > Any idea what could be wrong ? Or an idea how I could track down that issue ?
> >
> > Kind regards,
> >
> > Vincent