<div dir="ltr"><div dir="ltr">Hi,<br><br>I'm still stuck with the inability to stream h264 video to webrtcsink, but I've done a bit of research.<div><br>The issue is similar to this one albeit in slightly different context: <a href="https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/411">https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/411</a> "livekitwebrtcsink: failure to negotiate h264"<br>Starting from gstreamer's most obvious error log "There is no codec present that can handle the stream's type.", we get to webrtcsink's error "Error running discovery: No caps found for stream video_0" triggered by start_stream_discovery_if_needed (in webrtcsink\<a href="http://imp.rs">imp.rs</a>)<br>Increasing the log level, this error is preceeded by 4 times the following warning:<br>net\webrtc\src\webrtcsink\imp.rs:3226:gstrswebrtc::webrtcsink::imp::BaseWebRTCSink::lookup_caps::{{closure}}:<wsink> Codec discovery pipeline failed: Linking encoding elements<br>The message "Codec discovery pipeline failed" is logged from lookup_caps(), and if I understand this code correctly, there are two cases:<br>- either "Stream is already encoded with [a supported] codec", and it works out of the box without trying to look up caps (that's what happens when input is vp8)<br>- or the incoming stream uses another codec (h264 or mpeg2video in my tests), and the code tries to enumerate the caps, which fails.<br>Does this make sense ?<br><br>I'm willing to help (I have zero Rust experience but always eager to learn) and I was able to rebuild the plugin from the sources, but I don't know how to tell gstreamer to use my home-brewed webrtcsink instead of the one in the gstreamer distribution. <br>Is it statically linked in the gstreamer build ?<br><br>Any help would be greatly appreciated.<br><br>KR,<br><br>Vincent<br><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 17, 2023 at 1:13 AM Vincent Deconinck <<a href="mailto:vdeconinck@gmail.com">vdeconinck@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">Hi,<br><br>Thanks for all the insightful answers. The log suggestion and the audio frame timing issue triggered the idea of a few more tests.<br><br>Here are the tests I did today:<br>1. (set logging with GST_DEBUG=5). Ran the gstreamer server and the browser with "choppy audio". The log is 350+MB and file writing clearly impacts the playback (audio choppiness is increased and video becomes choppy too). AFAICT however, the log shows no ERROR until the end of the video ("pause task, reason: Flushing"). The zipped log is available on <a href="https://we.tl/t-n3kLKluEXE" target="_blank">https://we.tl/t-n3kLKluEXE</a><br>2. (still logging with GST_DEBUG=5) Changed the pipeline to only use the audio from the webm trailer video file, and used a videotestsrc as the video source => audio is still very choppy<br>3. (still logging with GST_DEBUG=5) Extracted the audio to a plain uncompressed wav file (obtained with "ffmpeg -i sintel_trailer-480p.webm sintel_trailer-480p.wav") and used this as the audio source and the videotestsrc as the video source => audio is still very choppy<br>4. (no logging from here on) Same test: WAV file as the audio source and the videotestsrc as the video source => audio is perfect !<br>5. Back to using the webm file as the audio source and the videotestsrc as the video source => audio is choppy<br>6. Converted the source file (vp8+vorbis) to mpeg4 (h264+aac) with "ffmpeg -i sintel_trailer-480p.webm sintel_trailer-480p.mp4" and used this as the audio source and the videotestsrc as the video source => audio is "almost" perfect (I can still hear a "drop" in the second drum at 00:16, which is not present when playing the mp4 file locally with VLC).<br>7. Used the mp4 as source for both video and audio => Fails with "Error received from element wsink: There is no codec present that can handle the stream's type.". Seems h264 video is not recognized ?<br>8. Converted the source file (vp8+vorbis) to a ts (mpeg2video+ac3) with "ffmpeg -i sintel_trailer-480p.webm -acodec ac3 -vcodec mpeg2video sintel_trailer-480p.ts" and used it for both video and audio => Fails with "There is no codec present that can handle the stream's type". Seems mpeg2video is also not recognized ?<br>9. Baked an alternate webm file using opus instead of vorbis with "ffmpeg -i sintel_trailer-480p.webm -strict -2 -acodec opus -vcodec copy sintel_trailer-480p_opus.webm" => audio/video is perfect !<br><br>OK. What did I learn ?<br>- a lack of resources (e.g. heavy logging) causes choppiness. My PC is quite good though (core i9 3.6GHz 32GB w/ GeForce RTX 2060) but maybe logging chokes gsteamer.<br>Regarding audio codecs :<br>- Vorbis causes atrocious choppiness.<br>- AAC is almost OK<br>- Wav and Opus are perfect<br>Regarding video codecs:<br>- It seems this pipeline only accepts vp8 as input<br><br>Questions: <br>My source files being either mpeg4 (with aac or ac3 audio) or even MXF (and conversion beforehand not being an option),<br>- How can I convert the videos on the fly ? <br>- Isn't vconvert meant to perform the conversion if required ?<br><br>Sorry again for the newbie questions...<br><br>Kind regards,<br><br>Vincent<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 16, 2023 at 7:07 PM cfd new <<a href="mailto:newcfd@yahoo.com" target="_blank">newcfd@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:13px"><div></div>
<div dir="ltr">Thank you so much for your detailed mail. Very nice!</div><div dir="ltr"><br></div><div dir="ltr"> Joe<br></div><div><br></div>
</div><div id="m_6592454021983077147m_345580310381850135m_5098608030515826616yahoo_quoted_0587936476">
<div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:13px;color:rgb(38,40,42)">
<div>
On Thursday, November 16, 2023, 01:00:44 p.m. EST, cfd new via gstreamer-devel <<a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>> wrote:
</div>
<div><br></div>
<div><br></div>
<div><div id="m_6592454021983077147m_345580310381850135m_5098608030515826616yiv6872810570"><div><div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:13px"><div></div>
<div dir="ltr">Hi, Philipp, <br clear="none"></div><div dir="ltr"><br clear="none"></div><div dir="ltr"> very interested in your code. I got the following message when I stream a live video to youtube. Maybe it is related to your case.</div><div dir="ltr"><br clear="none"></div><div dir="ltr"> Joe<br clear="none"></div><div dir="ltr"><div><pre><code><span>0</span><span>:</span><span>03</span><span>:</span><span>35.140304346</span> <span>21323</span> <span>0x7f1fa004c240</span> <span>WARN</span> audioresample gstaudioresample.<span>c:</span><span>732</span><span>:gst_audio_resample_check_discont</span><span>:<audioresample1></span> encountered timestamp discontinuity of <span>639</span> samples = <span>0</span><span>:</span><span>00</span><span>:</span><span>00</span>.039937500
<span>0</span><span>:</span><span>03</span><span>:</span><span>35.141001930</span> <span>21323</span> <span>0x7f1fa004c240</span> <span>WARN</span> audioresample gstaudioresample.<span>c:</span><span>732</span><span>:gst_audio_resample_check_discont</span><span>:<audioresample1></span> encountered timestamp discontinuity of <span>639</span> samples = <span>0</span><span>:</span><span>00</span><span>:</span><span>00</span>.039937500
<span>0</span><span>:</span><span>03</span><span>:</span><span>35.141314636</span> <span>21323</span> <span>0x7f1fa004c240</span> <span>WARN</span> audioresample gstaudioresample.<span>c:</span><span>732</span><span>:gst_audio_resample_check_discont</span><span>:<audioresample1></span> encountered timestamp discontinuity of <span>639</span> samples = <span>0</span><span>:</span><span>00</span><span>:</span><span>00</span>.039937500
</code></pre></div><div><br clear="none"></div></div>
</div><div id="m_6592454021983077147m_345580310381850135m_5098608030515826616yiv6872810570yqt75092"><div id="m_6592454021983077147m_345580310381850135m_5098608030515826616yiv6872810570yahoo_quoted_1151053006">
<div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:13px;color:rgb(38,40,42)">
<div>
On Thursday, November 16, 2023, 06:35:55 a.m. EST, Philipp B via gstreamer-devel <<a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>> wrote:
</div>
<div><br clear="none"></div>
<div><br clear="none"></div>
<div><div dir="ltr">Hi,<br clear="none"><br clear="none">[dislaimer: I didnt work with gstreamer for some time, my terminology<br clear="none">might be a bit rusty]<br clear="none"><br clear="none">I also had audio issues when streaming video files over webrtc, and<br clear="none">judging by your video, it could be the same issue you face.<br clear="none"><br clear="none">For me it turned out to be small rounding errors in the audio frame<br clear="none">timestamps. Due to the realtime nature, browsers playing webrtc audio<br clear="none">are not aiming for a perfect smooth playback, but rather for perfect<br clear="none">realtime. Small deviations in timestamps instantly lead to the browser<br clear="none">inserting small pieces of silence, or dropping a few ms of audio.<br clear="none"><br clear="none">The most obvious effect was the volume being way lower than it should<br clear="none">be. This was caused by the player/browser constantly fading segments<br clear="none">in and out, to overlap them with the next one, or with silence. Also,<br clear="none">I had similar distortion artifacts than you have.<br clear="none"><br clear="none">To be clear, my issue seemed to be related to transmission of<br clear="none">non-live, pre-encoded video only. WebRTC is made for live<br clear="none">transmissions, where audio frame timestamps are directly related to<br clear="none">the wall clock time at which a segment has been recorded. In contrast,<br clear="none">a non-live video file typically has a stream of audio segments, which<br clear="none">are expected to be played back without gaps, even when there are small<br clear="none">gaps according to the time stamps.<br clear="none"><br clear="none">What happens if you increase the audio frame size? In my case, this<br clear="none">was improving the situation dramatically, but not fully fixing it.<br clear="none"><br clear="none">In the end, I got this solved by "smoothing" audio time stamps. I<br clear="none">wrote my own gstreamer element, that kept an estimation of the next<br clear="none">frames timestamp, based on the last frame. In case, there is a small<br clear="none">deviation (e.g. below 5 ms), the frames timestamp will be changed to<br clear="none">that of the estimation. I can provide the code of that element in case<br clear="none">your interested.<br clear="none"><br clear="none">Philipp<br clear="none"><br clear="none">Am Mi., 15. Nov. 2023 um 23:39 Uhr schrieb Vincent Deconinck via<br clear="none">gstreamer-devel <<a rel="nofollow noopener noreferrer" shape="rect" href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a>>:<div id="m_6592454021983077147m_345580310381850135m_5098608030515826616yiv6872810570yqtfd11681"><br clear="none">><br clear="none">> Hi,<br clear="none">><br clear="none">> I made some progress in my quest to stream video files to a browser using webrtcssink, but audio is not OK. I built the following pipeline:<br clear="none">> +-----------------------------------------------------------------------------+<br clear="none">> | uridecodebin.audio ->- audioconvert ->- audioresample ->- webrtcssink.audio |<br clear="none">> | uridecodebin.video ----->----- videoconvert ------>------ webrtcssink.video |<br clear="none">> +-----------------------------------------------------------------------------+<br clear="none">><br clear="none">> The webrtcsink pads are statically linked, and the uridecodebin pads are dynamically linked upon "pad_added" signal (as in <a rel="nofollow noopener noreferrer" shape="rect" href="https://gstreamer.freedesktop.org/documentation/tutorials/basic/dynamic-pipelines.html" target="_blank">https://gstreamer.freedesktop.org/documentation/tutorials/basic/dynamic-pipelines.html</a> ).<br clear="none">><br clear="none">> Problem: The video looks OK in the browser, but the audio is choppy. A quick and dirty (sorry) video tells a thousand words: <a rel="nofollow noopener noreferrer" shape="rect" href="https://youtu.be/1EbPHLZvVLc?t=27" target="_blank">https://youtu.be/1EbPHLZvVLc?t=27</a><br clear="none">><br clear="none">> (using signalling server, JS API and gstreamer built 2 days ago from git HEAD)<br clear="none">><br clear="none">> I was first using the http URI to the source video but I also tried using a local copy of the file, to no avail.<br clear="none">><br clear="none">> Any idea what could be wrong ? Or an idea how I could track down that issue ?<br clear="none">><br clear="none">> Kind regards,<br clear="none">><br clear="none">> Vincent<br clear="none"></div></div></div>
</div>
</div></div></div></div></div>
</div>
</div></div></blockquote></div>
</div></div>
</blockquote></div></div>