Choppy audio playing a videofile through webrtcsink

Sun Nov 19 02:10:46 UTC 2023

Hi,

I'm still stuck with the inability to stream h264 video to webrtcsink, but
I've done a bit of research.

The issue is similar to this one albeit in slightly different context:
https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/411
"livekitwebrtcsink: failure to negotiate h264"
Starting from gstreamer's most obvious error log "There is no codec present
that can handle the stream's type.", we get to webrtcsink's error "Error
running discovery: No caps found for stream video_0" triggered by
start_stream_discovery_if_needed (in webrtcsink\imp.rs)
Increasing the log level, this error is preceeded by 4 times the following
warning:
net\webrtc\src\webrtcsink\imp.rs:3226:gstrswebrtc::webrtcsink::imp::BaseWebRTCSink::lookup_caps::{{closure}}:<wsink>
Codec discovery pipeline failed: Linking encoding elements
The message "Codec discovery pipeline failed" is logged from lookup_caps(),
and if I understand this code correctly, there are two cases:
- either "Stream is already encoded with [a supported] codec", and it works
out of the box without trying to look up caps (that's what happens when
input is vp8)
- or the incoming stream uses another codec (h264 or mpeg2video in my
tests), and the code tries to enumerate the caps, which fails.
Does this make sense ?

I'm willing to help (I have zero Rust experience but always eager to learn)
and I was able to rebuild the plugin from the sources, but I don't know how
to tell gstreamer to use my home-brewed webrtcsink instead of the one in
the gstreamer distribution.
Is it statically linked in the gstreamer build ?

Any help would be greatly appreciated.

KR,

Vincent

On Fri, Nov 17, 2023 at 1:13 AM Vincent Deconinck <vdeconinck at gmail.com>
wrote:

> Hi,
>
> Thanks for all the insightful answers. The log suggestion and the audio
> frame timing issue triggered the idea of a few more tests.
>
> Here are the tests I did today:
> 1. (set logging with GST_DEBUG=5). Ran the gstreamer server and the
> browser with "choppy audio". The log is 350+MB and file writing clearly
> impacts the playback (audio choppiness is increased and video becomes
> choppy too). AFAICT however, the log shows no ERROR until the end of the
> video ("pause task, reason: Flushing"). The zipped log is available on
> https://we.tl/t-n3kLKluEXE
> 2. (still logging with GST_DEBUG=5) Changed the pipeline to only use the
> audio from the webm trailer video file, and used a videotestsrc as the
> video source => audio is still very choppy
> 3. (still logging with GST_DEBUG=5) Extracted the audio to a plain
> uncompressed wav file (obtained with "ffmpeg -i sintel_trailer-480p.webm
> sintel_trailer-480p.wav") and used this as the audio source and the
> videotestsrc as the video source => audio is still very choppy
> 4. (no logging from here on) Same test: WAV file as the audio source and
> the videotestsrc as the video source => audio is perfect !
> 5. Back to using the webm file as the audio source and the videotestsrc as
> the video source => audio is choppy
> 6. Converted the source file (vp8+vorbis) to mpeg4 (h264+aac) with "ffmpeg
> -i sintel_trailer-480p.webm sintel_trailer-480p.mp4" and used this as the
> audio source and the videotestsrc as the video source => audio is "almost"
> perfect (I can still hear a "drop" in the second drum at 00:16, which is
> not present when playing the mp4 file locally with VLC).
> 7. Used the mp4 as source for both video and audio => Fails with "Error
> received from element wsink: There is no codec present that can handle the
> stream's type.". Seems h264 video is not recognized ?
> 8. Converted the source file (vp8+vorbis) to a ts (mpeg2video+ac3) with
> "ffmpeg -i sintel_trailer-480p.webm -acodec ac3 -vcodec mpeg2video
> sintel_trailer-480p.ts" and used it for both video and audio => Fails with
> "There is no codec present that can handle the stream's type".
> Seems mpeg2video is also not recognized ?
> 9. Baked an alternate webm file using opus instead of vorbis with "ffmpeg
> -i sintel_trailer-480p.webm -strict -2 -acodec opus -vcodec copy
> sintel_trailer-480p_opus.webm" => audio/video is perfect !
>
> OK. What did I learn ?
> - a lack of resources (e.g. heavy logging) causes choppiness. My PC is
> quite good though (core i9 3.6GHz 32GB w/ GeForce RTX 2060) but maybe
> logging chokes gsteamer.
> Regarding audio codecs :
> - Vorbis causes atrocious choppiness.
> - AAC is almost OK
> - Wav and Opus are perfect
> Regarding video codecs:
> - It seems this pipeline only accepts vp8 as input
>
> Questions:
> My source files being either mpeg4 (with aac or ac3 audio) or even MXF
> (and conversion beforehand not being an option),
> - How can I convert the videos on the fly ?
> - Isn't vconvert meant to perform the conversion if required ?
>
> Sorry again for the newbie questions...
>
> Kind regards,
>
> Vincent
>
> On Thu, Nov 16, 2023 at 7:07 PM cfd new <newcfd at yahoo.com> wrote:
>
>> Thank you so much for your detailed mail. Very nice!
>>
>>     Joe
>>
>> On Thursday, November 16, 2023, 01:00:44 p.m. EST, cfd new via
>> gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
>>
>>
>> Hi, Philipp,
>>
>>    very interested in your code. I got the following message when I
>> stream a live video to youtube. Maybe it is related to your case.
>>
>>    Joe
>>
>> 0:03:35.140304346 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.0399375000:03:35.141001930 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.0399375000:03:35.141314636 21323 0x7f1fa004c240 WARN           audioresample gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresample1> encountered timestamp discontinuity of 639 samples = 0:00:00.039937500
>>
>>
>> On Thursday, November 16, 2023, 06:35:55 a.m. EST, Philipp B via
>> gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
>>
>>
>> Hi,
>>
>> [dislaimer: I didnt work with gstreamer for some time, my terminology
>> might be a bit rusty]
>>
>> I also had audio issues when streaming video files over webrtc, and
>> judging by your video, it could be the same issue you face.
>>
>> For me it turned out to be small rounding errors in the audio frame
>> timestamps. Due to the realtime nature, browsers playing webrtc audio
>> are not aiming for a perfect smooth playback, but rather for perfect
>> realtime. Small deviations in timestamps instantly lead to the browser
>> inserting small pieces of silence, or dropping a few ms of audio.
>>
>> The most obvious effect was the volume being way lower than it should
>> be. This was caused by the player/browser constantly fading segments
>> in and out, to overlap them with the next one, or with silence. Also,
>> I had similar distortion artifacts than you have.
>>
>> To be clear, my issue seemed to be related to transmission of
>> non-live, pre-encoded video only. WebRTC is made for live
>> transmissions, where audio frame timestamps are directly related to
>> the wall clock time at which a segment has been recorded. In contrast,
>> a non-live video file typically has a stream of audio segments, which
>> are expected to be played back without gaps, even when there are small
>> gaps according to the time stamps.
>>
>> What happens if you increase the audio frame size? In my case, this
>> was improving the situation dramatically, but not fully fixing it.
>>
>> In the end, I got this solved by "smoothing" audio time stamps. I
>> wrote my own gstreamer element, that kept an estimation of the next
>> frames timestamp, based on the last frame. In case, there is a small
>> deviation (e.g. below 5 ms), the frames timestamp will be changed to
>> that of the estimation. I can provide the code of that element in case
>> your interested.
>>
>> Philipp
>>
>> Am Mi., 15. Nov. 2023 um 23:39 Uhr schrieb Vincent Deconinck via
>> gstreamer-devel <gstreamer-devel at lists.freedesktop.org>:
>>
>> >
>> > Hi,
>> >
>> > I made some progress in my quest to stream video files to a browser
>> using webrtcssink, but audio is not OK. I built the following pipeline:
>> >
>> +-----------------------------------------------------------------------------+
>> > | uridecodebin.audio ->- audioconvert ->- audioresample ->-
>> webrtcssink.audio |
>> > | uridecodebin.video ----->----- videoconvert ------>------
>> webrtcssink.video |
>> >
>> +-----------------------------------------------------------------------------+
>> >
>> > The webrtcsink pads are statically linked, and the uridecodebin pads
>> are dynamically linked upon "pad_added" signal (as in
>> https://gstreamer.freedesktop.org/documentation/tutorials/basic/dynamic-pipelines.html
>> ).
>> >
>> > Problem: The video looks OK in the browser, but the audio is choppy. A
>> quick and dirty (sorry) video tells a thousand words:
>> https://youtu.be/1EbPHLZvVLc?t=27
>> >
>> > (using signalling server, JS API and gstreamer built 2 days ago from
>> git HEAD)
>> >
>> > I was first using the http URI to the source video but I also tried
>> using a local copy of the file, to no avail.
>> >
>> > Any idea what could be wrong ? Or an idea how I could track down that
>> issue ?
>> >
>> > Kind regards,
>> >
>> > Vincent
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20231119/8d99161b/attachment.htm>