Choppy audio playing a videofile through webrtcsink

Fri Nov 17 00:13:55 UTC 2023

Hi,

Thanks for all the insightful answers. The log suggestion and the audio
frame timing issue triggered the idea of a few more tests.

Here are the tests I did today:
1. (set logging with GST_DEBUG=5). Ran the gstreamer server and the browser
with "choppy audio". The log is 350+MB and file writing clearly impacts the
playback (audio choppiness is increased and video becomes choppy too).
AFAICT however, the log shows no ERROR until the end of the video ("pause
task, reason: Flushing"). The zipped log is available on
https://we.tl/t-n3kLKluEXE
2. (still logging with GST_DEBUG=5) Changed the pipeline to only use the
audio from the webm trailer video file, and used a videotestsrc as the
video source => audio is still very choppy
3. (still logging with GST_DEBUG=5) Extracted the audio to a plain
uncompressed wav file (obtained with "ffmpeg -i sintel_trailer-480p.webm
sintel_trailer-480p.wav") and used this as the audio source and the
videotestsrc as the video source => audio is still very choppy
4. (no logging from here on) Same test: WAV file as the audio source and
the videotestsrc as the video source => audio is perfect !
5. Back to using the webm file as the audio source and the videotestsrc as
the video source => audio is choppy
6. Converted the source file (vp8+vorbis) to mpeg4 (h264+aac) with "ffmpeg
-i sintel_trailer-480p.webm sintel_trailer-480p.mp4" and used this as the
audio source and the videotestsrc as the video source => audio is "almost"
perfect (I can still hear a "drop" in the second drum at 00:16, which is
not present when playing the mp4 file locally with VLC).
7. Used the mp4 as source for both video and audio => Fails with "Error
received from element wsink: There is no codec present that can handle the
stream's type.". Seems h264 video is not recognized ?
8. Converted the source file (vp8+vorbis) to a ts (mpeg2video+ac3) with
"ffmpeg -i sintel_trailer-480p.webm -acodec ac3 -vcodec mpeg2video
sintel_trailer-480p.ts" and used it for both video and audio => Fails with
"There is no codec present that can handle the stream's type".
Seems mpeg2video is also not recognized ?
9. Baked an alternate webm file using opus instead of vorbis with "ffmpeg
-i sintel_trailer-480p.webm -strict -2 -acodec opus -vcodec copy
sintel_trailer-480p_opus.webm" => audio/video is perfect !

OK. What did I learn ?
- a lack of resources (e.g. heavy logging) causes choppiness. My PC is
quite good though (core i9 3.6GHz 32GB w/ GeForce RTX 2060) but maybe
logging chokes gsteamer.
Regarding audio codecs :
- Vorbis causes atrocious choppiness.
- AAC is almost OK
- Wav and Opus are perfect
Regarding video codecs:
- It seems this pipeline only accepts vp8 as input

Questions:
My source files being either mpeg4 (with aac or ac3 audio) or even MXF (and
conversion beforehand not being an option),
- How can I convert the videos on the fly ?
- Isn't vconvert meant to perform the conversion if required ?

Sorry again for the newbie questions...

Kind regards,

Vincent

On Thu, Nov 16, 2023 at 7:07 PM cfd new <newcfd at yahoo.com> wrote:

> Thank you so much for your detailed mail. Very nice!
>
>     Joe
>
> On Thursday, November 16, 2023, 01:00:44 p.m. EST, cfd new via
> gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
>
>
> Hi, Philipp,
>
>    very interested in your code. I got the following message when I stream
> a live video to youtube. Maybe it is related to your case.
>
>    Joe
>
> 0:03:35.140304346 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.0399375000:03:35.141001930 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.0399375000:03:35.141314636 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.039937500
>
>
> On Thursday, November 16, 2023, 06:35:55 a.m. EST, Philipp B via
> gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
>
>
> Hi,
>
> [dislaimer: I didnt work with gstreamer for some time, my terminology
> might be a bit rusty]
>
> I also had audio issues when streaming video files over webrtc, and
> judging by your video, it could be the same issue you face.
>
> For me it turned out to be small rounding errors in the audio frame
> timestamps. Due to the realtime nature, browsers playing webrtc audio
> are not aiming for a perfect smooth playback, but rather for perfect
> realtime. Small deviations in timestamps instantly lead to the browser
> inserting small pieces of silence, or dropping a few ms of audio.
>
> The most obvious effect was the volume being way lower than it should
> be. This was caused by the player/browser constantly fading segments
> in and out, to overlap them with the next one, or with silence. Also,
> I had similar distortion artifacts than you have.
>
> To be clear, my issue seemed to be related to transmission of
> non-live, pre-encoded video only. WebRTC is made for live
> transmissions, where audio frame timestamps are directly related to
> the wall clock time at which a segment has been recorded. In contrast,
> a non-live video file typically has a stream of audio segments, which
> are expected to be played back without gaps, even when there are small
> gaps according to the time stamps.
>
> What happens if you increase the audio frame size? In my case, this
> was improving the situation dramatically, but not fully fixing it.
>
> In the end, I got this solved by "smoothing" audio time stamps. I
> wrote my own gstreamer element, that kept an estimation of the next
> frames timestamp, based on the last frame. In case, there is a small
> deviation (e.g. below 5 ms), the frames timestamp will be changed to
> that of the estimation. I can provide the code of that element in case
> your interested.
>
> Philipp
>
> Am Mi., 15. Nov. 2023 um 23:39 Uhr schrieb Vincent Deconinck via
> gstreamer-devel <gstreamer-devel at lists.freedesktop.org>:
>
> >
> > Hi,
> >
> > I made some progress in my quest to stream video files to a browser
> using webrtcssink, but audio is not OK. I built the following pipeline:
> >
> +-----------------------------------------------------------------------------+
> > | uridecodebin.audio ->- audioconvert ->- audioresample ->-
> webrtcssink.audio |
> > | uridecodebin.video ----->----- videoconvert ------>------
> webrtcssink.video |
> >
> +-----------------------------------------------------------------------------+
> >
> > The webrtcsink pads are statically linked, and the uridecodebin pads are
> dynamically linked upon "pad_added" signal (as in
> https://gstreamer.freedesktop.org/documentation/tutorials/basic/dynamic-pipelines.html
> ).
> >
> > Problem: The video looks OK in the browser, but the audio is choppy. A
> quick and dirty (sorry) video tells a thousand words:
> https://youtu.be/1EbPHLZvVLc?t=27
> >
> > (using signalling server, JS API and gstreamer built 2 days ago from git
> HEAD)
> >
> > I was first using the http URI to the source video but I also tried
> using a local copy of the file, to no avail.
> >
> > Any idea what could be wrong ? Or an idea how I could track down that
> issue ?
> >
> > Kind regards,
> >
> > Vincent
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20231117/b5be0917/attachment.htm>