<div dir="ltr"><div dir="ltr">Hi Mathieu,<div> </div><div>Thanks again for your replies.</div><div>As uridecodebin already includes queues (if I read correctly), I tried adding A/V queues (with default values) at the end, as follows : </div><div>+---------------------------------------------------------------------------------------+ | uridecodebin.audio ->- audioconvert ->- audioresample ->- queue ->- webrtcssink.audio |</div><div>| uridecodebin.video ----->----- videoconvert ------>------ queue ->- webrtcssink.video |</div><div>+---------------------------------------------------------------------------------------+ </div></div><div class="gmail_quote"><div class="gmail_attr">But unfortunately it did not bring any audible improvement.</div><div class="gmail_attr">I also tried adding another one between converting and resampling audio, as follows:</div><div class="gmail_attr"><div dir="ltr"><div>+-------------------------------------------------------------------------------------------------+ | uridecodebin.audio ->- audioconvert ->- queue ->- audioresample ->- queue ->- webrtcssink.audio |</div><div>| uridecodebin.video -------->-------- videoconvert -------->-------- queue ->- webrtcssink.video |</div><div>+-------------------------------------------------------------------------------------------------+ </div></div><div class="gmail_quote"><div class="gmail_attr">To no avail :-(</div><div class="gmail_attr"> </div></div></div><div class="gmail_attr">But maybe that wasn't what you meant ? </div><div class="gmail_attr"> </div><div class="gmail_attr">KR, </div><div class="gmail_attr"> </div><div class="gmail_attr">Vincent</div><div class="gmail_attr"> </div><div dir="ltr" class="gmail_attr">On Mon, Nov 20, 2023 at 3:05 PM Mathieu Duponchelle <<a href="mailto:mathieu@centricular.com" target="_blank">mathieu@centricular.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Have you tried using queues to smooth things out between the uridecodebin and webrtcsink? On Fri, 2023-11-17 at 01:13 +0100, Vincent Deconinck via gstreamer- devel wrote: > Hi, > > Thanks for all the insightful answers. The log suggestion and the > audio frame timing issue triggered the idea of a few more tests. > > Here are the tests I did today: > 1. (set logging with GST_DEBUG=5). Ran the gstreamer server and the > browser with "choppy audio". The log is 350+MB and file writing > clearly impacts the playback (audio choppiness is increased and video > becomes choppy too). AFAICT however, the log shows no ERROR until the > end of the video ("pause task, reason: Flushing"). The zipped log is > available on <a href="https://we.tl/t-n3kLKluEXE" rel="noreferrer" target="_blank">https://we.tl/t-n3kLKluEXE</a> > 2. (still logging with GST_DEBUG=5) Changed the pipeline to only use > the audio from the webm trailer video file, and used a videotestsrc > as the video source => audio is still very choppy > 3. (still logging with GST_DEBUG=5) Extracted the audio to a plain > uncompressed wav file (obtained with "ffmpeg -i sintel_trailer- > 480p.webm sintel_trailer-480p.wav") and used this as the audio source > and the videotestsrc as the video source => audio is still very > choppy > 4. (no logging from here on) Same test: WAV file as the audio source > and the videotestsrc as the video source => audio is perfect ! > 5. Back to using the webm file as the audio source and the > videotestsrc as the video source => audio is choppy > 6. Converted the source file (vp8+vorbis) to mpeg4 (h264+aac) with > "ffmpeg -i sintel_trailer-480p.webm sintel_trailer-480p.mp4" and used > this as the audio source and the videotestsrc as the video source => > audio is "almost" perfect (I can still hear a "drop" in the second > drum at 00:16, which is not present when playing the mp4 file locally > with VLC). > 7. Used the mp4 as source for both video and audio => Fails with > "Error received from element wsink: There is no codec present that > can handle the stream's type.". Seems h264 video is not recognized ? > 8. Converted the source file (vp8+vorbis) to a ts (mpeg2video+ac3) > with "ffmpeg -i sintel_trailer-480p.webm -acodec ac3 -vcodec > mpeg2video sintel_trailer-480p.ts" and used it for both video and > audio => Fails with "There is no codec present that can handle the > stream's type". Seems mpeg2video is also not recognized ? > 9. Baked an alternate webm file using opus instead of vorbis with > "ffmpeg -i sintel_trailer-480p.webm -strict -2 -acodec opus -vcodec > copy sintel_trailer-480p_opus.webm" => audio/video is perfect ! > > OK. What did I learn ? > - a lack of resources (e.g. heavy logging) causes choppiness. My PC > is quite good though (core i9 3.6GHz 32GB w/ GeForce RTX 2060) but > maybe logging chokes gsteamer. > Regarding audio codecs : > - Vorbis causes atrocious choppiness. > - AAC is almost OK > - Wav and Opus are perfect > Regarding video codecs: > - It seems this pipeline only accepts vp8 as input > > Questions: > My source files being either mpeg4 (with aac or ac3 audio) or even > MXF (and conversion beforehand not being an option), > - How can I convert the videos on the fly ? > - Isn't vconvert meant to perform the conversion if required ? > > Sorry again for the newbie questions... > > Kind regards, > > Vincent > > On Thu, Nov 16, 2023 at 7:07 PM cfd new <<a href="mailto:newcfd@yahoo.com" target="_blank">newcfd@yahoo.com</a>> wrote: > > > > Thank you so much for your detailed mail. Very nice! > > > > Joe > > > > > > > > > > On Thursday, November 16, 2023, 01:00:44 p.m. EST, cfd new via > > gstreamer-devel <<a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>> wrote: > > > > > > > > > > > > > > Hi, Philipp, > > > > very interested in your code. I got the following message when I > > stream a live video to youtube. Maybe it is related to your case. > > > > Joe > > 0:03:35.140304346 21323 0x7f1fa004c240 WARN audioresample > > gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresam > > ple1> encountered timestamp discontinuity of 639 samples = > > 0:00:00.039937500 > > 0:03:35.141001930 21323 0x7f1fa004c240 WARN audioresample > > gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresam > > ple1> encountered timestamp discontinuity of 639 samples = > > 0:00:00.039937500 > > 0:03:35.141314636 21323 0x7f1fa004c240 WARN audioresample > > gstaudioresample.c:732:gst_audio_resample_check_discont:<audioresam > > ple1> encountered timestamp discontinuity of 639 samples = > > 0:00:00.039937500 > > > > > > > > > > On Thursday, November 16, 2023, 06:35:55 a.m. EST, Philipp B via > > gstreamer-devel <<a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>> wrote: > > > > > > > > > > > > Hi, > > > > [dislaimer: I didnt work with gstreamer for some time, my > > terminology > > might be a bit rusty] > > > > I also had audio issues when streaming video files over webrtc, and > > judging by your video, it could be the same issue you face. > > > > For me it turned out to be small rounding errors in the audio frame > > timestamps. Due to the realtime nature, browsers playing webrtc > > audio > > are not aiming for a perfect smooth playback, but rather for > > perfect > > realtime. Small deviations in timestamps instantly lead to the > > browser > > inserting small pieces of silence, or dropping a few ms of audio. > > > > The most obvious effect was the volume being way lower than it > > should > > be. This was caused by the player/browser constantly fading > > segments > > in and out, to overlap them with the next one, or with silence. > > Also, > > I had similar distortion artifacts than you have. > > > > To be clear, my issue seemed to be related to transmission of > > non-live, pre-encoded video only. WebRTC is made for live > > transmissions, where audio frame timestamps are directly related to > > the wall clock time at which a segment has been recorded. In > > contrast, > > a non-live video file typically has a stream of audio segments, > > which > > are expected to be played back without gaps, even when there are > > small > > gaps according to the time stamps. > > > > What happens if you increase the audio frame size? In my case, this > > was improving the situation dramatically, but not fully fixing it. > > > > In the end, I got this solved by "smoothing" audio time stamps. I > > wrote my own gstreamer element, that kept an estimation of the next > > frames timestamp, based on the last frame. In case, there is a > > small > > deviation (e.g. below 5 ms), the frames timestamp will be changed > > to > > that of the estimation. I can provide the code of that element in > > case > > your interested. > > > > Philipp > > > > Am Mi., 15. Nov. 2023 um 23:39 Uhr schrieb Vincent Deconinck via > > gstreamer-devel <<a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>>: > > > > > > > > Hi, > > > > > > I made some progress in my quest to stream video files to a > > > browser using webrtcssink, but audio is not OK. I built the > > > following pipeline: > > > +---------------------------------------------------------------- > > > -------------+ > > > | uridecodebin.audio ->- audioconvert ->- audioresample ->- > > > webrtcssink.audio | > > > | uridecodebin.video ----->----- videoconvert ------>------ > > > webrtcssink.video | > > > +---------------------------------------------------------------- > > > -------------+ > > > > > > The webrtcsink pads are statically linked, and the uridecodebin > > > pads are dynamically linked upon "pad_added" signal (as in > > > <a href="https://gstreamer.freedesktop.org/documentation/tutorials/basic/dynamic-pipelines.html" rel="noreferrer" target="_blank">https://gstreamer.freedesktop.org/documentation/tutorials/basic/dynamic-pipelines.html</a> > > > ). > > > > > > Problem: The video looks OK in the browser, but the audio is > > > choppy. A quick and dirty (sorry) video tells a thousand words: > > > <a href="https://youtu.be/1EbPHLZvVLc?t=27" rel="noreferrer" target="_blank">https://youtu.be/1EbPHLZvVLc?t=27</a> > > > > > > (using signalling server, JS API and gstreamer built 2 days ago > > > from git HEAD) > > > > > > I was first using the http URI to the source video but I also > > > tried using a local copy of the file, to no avail. > > > > > > Any idea what could be wrong ? Or an idea how I could track down > > > that issue ? > > > > > > Kind regards, > > > > > > Vincent > > > > > > > > </blockquote></div></div>