Choppy audio playing a videofile through webrtcsink

Mon Nov 20 14:05:24 UTC 2023

Have you tried using queues to smooth things out between the
uridecodebin and webrtcsink?

On Fri, 2023-11-17 at 01:13 +0100, Vincent Deconinck via gstreamer-
devel wrote:
> Hi,
> 
> Thanks for all the insightful answers. The log suggestion and the
> audio frame timing issue triggered the idea of a few more tests.
> 
> Here are the tests I did today:
> 1. (set logging with GST_DEBUG=5). Ran the gstreamer server and the
> browser with "choppy audio". The log is 350+MB and file writing
> clearly impacts the playback (audio choppiness is increased and video
> becomes choppy too). AFAICT however, the log shows no ERROR until the
> end of the video ("pause task, reason: Flushing"). The zipped log is
> available on https://we.tl/t-n3kLKluEXE
> 2. (still logging with GST_DEBUG=5) Changed the pipeline to only use
> the audio from the webm trailer video file, and used a videotestsrc
> as the video source => audio is still very choppy
> 3. (still logging with GST_DEBUG=5) Extracted the audio to a plain
> uncompressed wav file (obtained with "ffmpeg -i sintel_trailer-
> 480p.webm sintel_trailer-480p.wav") and used this as the audio source
> and the videotestsrc as the video source => audio is still very
> choppy
> 4. (no logging from here on) Same test: WAV file as the audio source
> and the videotestsrc as the video source => audio is perfect !
> 5. Back to using the webm file as the audio source and the
> videotestsrc as the video source => audio is choppy
> 6. Converted the source file (vp8+vorbis) to mpeg4 (h264+aac) with
> "ffmpeg -i sintel_trailer-480p.webm sintel_trailer-480p.mp4" and used
> this as the audio source and the videotestsrc as the video source =>
> audio is "almost" perfect (I can still hear a "drop" in the second
> drum at 00:16, which is not present when playing the mp4 file locally
> with VLC).
> 7. Used the mp4 as source for both video and audio => Fails with
> "Error received from element wsink: There is no codec present that
> can handle the stream's type.". Seems h264 video is not recognized ?
> 8. Converted the source file (vp8+vorbis) to a ts (mpeg2video+ac3)
> with "ffmpeg -i sintel_trailer-480p.webm -acodec ac3 -vcodec
> mpeg2video sintel_trailer-480p.ts" and used it for both video and
> audio => Fails with "There is no codec present that can handle the
> stream's type". Seems mpeg2video is also not recognized ?
> 9. Baked an alternate webm file using opus instead of vorbis with
> "ffmpeg -i sintel_trailer-480p.webm -strict -2 -acodec opus -vcodec
> copy sintel_trailer-480p_opus.webm" => audio/video is perfect !
> 
> OK. What did I learn ?
> - a lack of resources (e.g. heavy logging) causes choppiness. My PC
> is quite good though (core i9 3.6GHz 32GB w/ GeForce RTX 2060) but
> maybe logging chokes gsteamer.
> Regarding audio codecs :
> - Vorbis causes atrocious choppiness.
> - AAC is almost OK
> - Wav and Opus are perfect
> Regarding video codecs:
> - It seems this pipeline only accepts vp8 as input
> 
> Questions: 
> My source files being either mpeg4 (with aac or ac3 audio) or even
> MXF (and conversion beforehand not being an option),
> - How can I convert the videos on the fly ? 
> - Isn't vconvert meant to perform the conversion if required ?
> 
> Sorry again for the newbie questions...
> 
> Kind regards,
> 
> Vincent
> 
> On Thu, Nov 16, 2023 at 7:07 PM cfd new <newcfd at yahoo.com> wrote:
> >  
> > Thank you so much for your detailed mail. Very nice!
> > 
> >     Joe
> > 
> >  
> >  
> >  
> >  On Thursday, November 16, 2023, 01:00:44 p.m. EST, cfd new via
> > gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote: 
> >  
> > 
> >  
> > 
> >  
> >  
> > Hi, Philipp, 
> > 
> >    very interested in your code. I got the following message when I
> > stream a live video to youtube. Maybe it is related to your case.
> > 
> >    Joe
> > 0:03:35.140304346 21323 0x7f1fa004c240 WARN           audioresample
> > gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresam
> > ple1> encountered timestamp discontinuity of 639 samples =
> > 0:00:00.039937500
> > 0:03:35.141001930 21323 0x7f1fa004c240 WARN           audioresample
> > gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresam
> > ple1> encountered timestamp discontinuity of 639 samples =
> > 0:00:00.039937500
> > 0:03:35.141314636 21323 0x7f1fa004c240 WARN           audioresample
> > gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresam
> > ple1> encountered timestamp discontinuity of 639 samples =
> > 0:00:00.039937500
> > 
> >  
> >  
> >  
> >  On Thursday, November 16, 2023, 06:35:55 a.m. EST, Philipp B via
> > gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote: 
> >  
> > 
> >  
> > 
> >  
> > Hi,
> > 
> > [dislaimer: I didnt work with gstreamer for some time, my
> > terminology
> > might be a bit rusty]
> > 
> > I also had audio issues when streaming video files over webrtc, and
> > judging by your video, it could be the same issue you face.
> > 
> > For me it turned out to be small rounding errors in the audio frame
> > timestamps. Due to the realtime nature, browsers playing webrtc
> > audio
> > are not aiming for a perfect smooth playback, but rather for
> > perfect
> > realtime. Small deviations in timestamps instantly lead to the
> > browser
> > inserting small pieces of silence, or dropping a few ms of audio.
> > 
> > The most obvious effect was the volume being way lower than it
> > should
> > be. This was caused by the player/browser constantly fading
> > segments
> > in and out, to overlap them with the next one, or with silence.
> > Also,
> > I had similar distortion artifacts than you have.
> > 
> > To be clear, my issue seemed to be related to transmission of
> > non-live, pre-encoded video only. WebRTC is made for live
> > transmissions, where audio frame timestamps are directly related to
> > the wall clock time at which a segment has been recorded. In
> > contrast,
> > a non-live video file typically has a stream of audio segments,
> > which
> > are expected to be played back without gaps, even when there are
> > small
> > gaps according to the time stamps.
> > 
> > What happens if you increase the audio frame size? In my case, this
> > was improving the situation dramatically, but not fully fixing it.
> > 
> > In the end, I got this solved by "smoothing" audio time stamps. I
> > wrote my own gstreamer element, that kept an estimation of the next
> > frames timestamp, based on the last frame. In case, there is a
> > small
> > deviation (e.g. below 5 ms), the frames timestamp will be changed
> > to
> > that of the estimation. I can provide the code of that element in
> > case
> > your interested.
> > 
> > Philipp
> > 
> > Am Mi., 15. Nov. 2023 um 23:39 Uhr schrieb Vincent Deconinck via
> > gstreamer-devel <gstreamer-devel at lists.freedesktop.org>:
> > 
> > > 
> > > Hi,
> > > 
> > > I made some progress in my quest to stream video files to a
> > > browser using webrtcssink, but audio is not OK. I built the
> > > following pipeline:
> > > +----------------------------------------------------------------
> > > -------------+
> > > | uridecodebin.audio ->- audioconvert ->- audioresample ->-
> > > webrtcssink.audio |
> > > | uridecodebin.video ----->----- videoconvert ------>------
> > > webrtcssink.video |
> > > +----------------------------------------------------------------
> > > -------------+
> > > 
> > > The webrtcsink pads are statically linked, and the uridecodebin
> > > pads are dynamically linked upon "pad_added" signal (as in
> > > https://gstreamer.freedesktop.org/documentation/tutorials/basic/dynamic-pipelines.html
> > > ).
> > > 
> > > Problem: The video looks OK in the browser, but the audio is
> > > choppy. A quick and dirty (sorry) video tells a thousand words:
> > > https://youtu.be/1EbPHLZvVLc?t=27
> > > 
> > > (using signalling server, JS API and gstreamer built 2 days ago
> > > from git HEAD)
> > > 
> > > I was first using the http URI to the source video but I also
> > > tried using a local copy of the file, to no avail.
> > > 
> > > Any idea what could be wrong ? Or an idea how I could track down
> > > that issue ?
> > > 
> > > Kind regards,
> > > 
> > > Vincent
> >  
> >  
> >  
> >  