<div dir="ltr"><div class="gmail_quote"><div dir="ltr">Hi,<br><br>Thanks for all the insightful answers. The log suggestion and the audio frame timing issue triggered the idea of a few more tests.<br><br>Here are the tests I did today:<br>1. (set logging with GST_DEBUG=5). Ran the gstreamer server and the browser with "choppy audio". The log is 350+MB and file writing clearly impacts the playback (audio choppiness is increased and video becomes choppy too). AFAICT however, the log shows no ERROR until the end of the video ("pause task, reason: Flushing"). The zipped log is available on <a href="https://we.tl/t-n3kLKluEXE" target="_blank">https://we.tl/t-n3kLKluEXE</a><br>2. (still logging with GST_DEBUG=5) Changed the pipeline to only use the audio from the webm trailer video file, and used a videotestsrc as the video source => audio is still very choppy<br>3. (still logging with GST_DEBUG=5) Extracted the audio to a plain uncompressed wav file (obtained with "ffmpeg -i sintel_trailer-480p.webm sintel_trailer-480p.wav") and used this as the audio source and the videotestsrc as the video source => audio is still very choppy<br>4. (no logging from here on) Same test: WAV file as the audio source and the videotestsrc as the video source => audio is perfect !<br>5. Back to using the webm file as the audio source and the videotestsrc as the video source => audio is choppy<br>6. Converted the source file (vp8+vorbis) to mpeg4 (h264+aac) with "ffmpeg -i sintel_trailer-480p.webm sintel_trailer-480p.mp4" and used this as the audio source and the videotestsrc as the video source => audio is "almost" perfect (I can still hear a "drop" in the second drum at 00:16, which is not present when playing the mp4 file locally with VLC).<br>7. Used the mp4 as source for both video and audio => Fails with "Error received from element wsink: There is no codec present that can handle the stream's type.". Seems h264 video is not recognized ?<br>8. Converted the source file (vp8+vorbis) to a ts (mpeg2video+ac3) with "ffmpeg -i sintel_trailer-480p.webm -acodec ac3 -vcodec mpeg2video sintel_trailer-480p.ts" and used it for both video and audio => Fails with "There is no codec present that can handle the stream's type". Seems mpeg2video is also not recognized ?<br>9. Baked an alternate webm file using opus instead of vorbis with "ffmpeg -i sintel_trailer-480p.webm -strict -2 -acodec opus -vcodec copy sintel_trailer-480p_opus.webm" => audio/video is perfect !<br><br>OK. What did I learn ?<br>- a lack of resources (e.g. heavy logging) causes choppiness. My PC is quite good though (core i9 3.6GHz 32GB w/ GeForce RTX 2060) but maybe logging chokes gsteamer.<br>Regarding audio codecs :<br>- Vorbis causes atrocious choppiness.<br>- AAC is almost OK<br>- Wav and Opus are perfect<br>Regarding video codecs:<br>- It seems this pipeline only accepts vp8 as input<br><br>Questions: <br>My source files being either mpeg4 (with aac or ac3 audio) or even MXF (and conversion beforehand not being an option),<br>- How can I convert the videos on the fly ? <br>- Isn't vconvert meant to perform the conversion if required ?<br><br>Sorry again for the newbie questions...<br><br>Kind regards,<br><br>Vincent<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 16, 2023 at 7:07 PM cfd new <<a href="mailto:newcfd@yahoo.com" target="_blank">newcfd@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:13px"><div></div>
<div dir="ltr">Thank you so much for your detailed mail. Very nice!</div><div dir="ltr"><br></div><div dir="ltr"> Joe<br></div><div><br></div>
</div><div id="m_345580310381850135m_5098608030515826616yahoo_quoted_0587936476">
<div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:13px;color:rgb(38,40,42)">
<div>
On Thursday, November 16, 2023, 01:00:44 p.m. EST, cfd new via gstreamer-devel <<a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>> wrote:
</div>
<div><br></div>
<div><br></div>
<div><div id="m_345580310381850135m_5098608030515826616yiv6872810570"><div><div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:13px"><div></div>
<div dir="ltr">Hi, Philipp, <br clear="none"></div><div dir="ltr"><br clear="none"></div><div dir="ltr"> very interested in your code. I got the following message when I stream a live video to youtube. Maybe it is related to your case.</div><div dir="ltr"><br clear="none"></div><div dir="ltr"> Joe<br clear="none"></div><div dir="ltr"><div><pre><code><span>0</span><span>:</span><span>03</span><span>:</span><span>35.140304346</span> <span>21323</span> <span>0x7f1fa004c240</span> <span>WARN</span> audioresample gstaudioresample.<span>c:</span><span>732</span><span>:gst_audio_resample_check_discont</span><span>:<audioresample1></span> encountered timestamp discontinuity of <span>639</span> samples = <span>0</span><span>:</span><span>00</span><span>:</span><span>00</span>.039937500
<span>0</span><span>:</span><span>03</span><span>:</span><span>35.141001930</span> <span>21323</span> <span>0x7f1fa004c240</span> <span>WARN</span> audioresample gstaudioresample.<span>c:</span><span>732</span><span>:gst_audio_resample_check_discont</span><span>:<audioresample1></span> encountered timestamp discontinuity of <span>639</span> samples = <span>0</span><span>:</span><span>00</span><span>:</span><span>00</span>.039937500
<span>0</span><span>:</span><span>03</span><span>:</span><span>35.141314636</span> <span>21323</span> <span>0x7f1fa004c240</span> <span>WARN</span> audioresample gstaudioresample.<span>c:</span><span>732</span><span>:gst_audio_resample_check_discont</span><span>:<audioresample1></span> encountered timestamp discontinuity of <span>639</span> samples = <span>0</span><span>:</span><span>00</span><span>:</span><span>00</span>.039937500
</code></pre></div><div><br clear="none"></div></div>
</div><div id="m_345580310381850135m_5098608030515826616yiv6872810570yqt75092"><div id="m_345580310381850135m_5098608030515826616yiv6872810570yahoo_quoted_1151053006">
<div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:13px;color:rgb(38,40,42)">
<div>
On Thursday, November 16, 2023, 06:35:55 a.m. EST, Philipp B via gstreamer-devel <<a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>> wrote:
</div>
<div><br clear="none"></div>
<div><br clear="none"></div>
<div><div dir="ltr">Hi,<br clear="none"><br clear="none">[dislaimer: I didnt work with gstreamer for some time, my terminology<br clear="none">might be a bit rusty]<br clear="none"><br clear="none">I also had audio issues when streaming video files over webrtc, and<br clear="none">judging by your video, it could be the same issue you face.<br clear="none"><br clear="none">For me it turned out to be small rounding errors in the audio frame<br clear="none">timestamps. Due to the realtime nature, browsers playing webrtc audio<br clear="none">are not aiming for a perfect smooth playback, but rather for perfect<br clear="none">realtime. Small deviations in timestamps instantly lead to the browser<br clear="none">inserting small pieces of silence, or dropping a few ms of audio.<br clear="none"><br clear="none">The most obvious effect was the volume being way lower than it should<br clear="none">be. This was caused by the player/browser constantly fading segments<br clear="none">in and out, to overlap them with the next one, or with silence. Also,<br clear="none">I had similar distortion artifacts than you have.<br clear="none"><br clear="none">To be clear, my issue seemed to be related to transmission of<br clear="none">non-live, pre-encoded video only. WebRTC is made for live<br clear="none">transmissions, where audio frame timestamps are directly related to<br clear="none">the wall clock time at which a segment has been recorded. In contrast,<br clear="none">a non-live video file typically has a stream of audio segments, which<br clear="none">are expected to be played back without gaps, even when there are small<br clear="none">gaps according to the time stamps.<br clear="none"><br clear="none">What happens if you increase the audio frame size? In my case, this<br clear="none">was improving the situation dramatically, but not fully fixing it.<br clear="none"><br clear="none">In the end, I got this solved by "smoothing" audio time stamps. I<br clear="none">wrote my own gstreamer element, that kept an estimation of the next<br clear="none">frames timestamp, based on the last frame. In case, there is a small<br clear="none">deviation (e.g. below 5 ms), the frames timestamp will be changed to<br clear="none">that of the estimation. I can provide the code of that element in case<br clear="none">your interested.<br clear="none"><br clear="none">Philipp<br clear="none"><br clear="none">Am Mi., 15. Nov. 2023 um 23:39 Uhr schrieb Vincent Deconinck via<br clear="none">gstreamer-devel <<a rel="nofollow noopener noreferrer" shape="rect" href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>>:<div id="m_345580310381850135m_5098608030515826616yiv6872810570yqtfd11681"><br clear="none">><br clear="none">> Hi,<br clear="none">><br clear="none">> I made some progress in my quest to stream video files to a browser using webrtcssink, but audio is not OK. I built the following pipeline:<br clear="none">> +-----------------------------------------------------------------------------+<br clear="none">> | uridecodebin.audio ->- audioconvert ->- audioresample ->- webrtcssink.audio |<br clear="none">> | uridecodebin.video ----->----- videoconvert ------>------ webrtcssink.video |<br clear="none">> +-----------------------------------------------------------------------------+<br clear="none">><br clear="none">> The webrtcsink pads are statically linked, and the uridecodebin pads are dynamically linked upon "pad_added" signal (as in <a rel="nofollow noopener noreferrer" shape="rect" href="https://gstreamer.freedesktop.org/documentation/tutorials/basic/dynamic-pipelines.html" target="_blank">https://gstreamer.freedesktop.org/documentation/tutorials/basic/dynamic-pipelines.html</a> ).<br clear="none">><br clear="none">> Problem: The video looks OK in the browser, but the audio is choppy. A quick and dirty (sorry) video tells a thousand words: <a rel="nofollow noopener noreferrer" shape="rect" href="https://youtu.be/1EbPHLZvVLc?t=27" target="_blank">https://youtu.be/1EbPHLZvVLc?t=27</a><br clear="none">><br clear="none">> (using signalling server, JS API and gstreamer built 2 days ago from git HEAD)<br clear="none">><br clear="none">> I was first using the http URI to the source video but I also tried using a local copy of the file, to no avail.<br clear="none">><br clear="none">> Any idea what could be wrong ? Or an idea how I could track down that issue ?<br clear="none">><br clear="none">> Kind regards,<br clear="none">><br clear="none">> Vincent<br clear="none"></div></div></div>
</div>
</div></div></div></div></div>
</div>
</div></div></blockquote></div>
</div></div>