<html><head></head><body><div class="ydp12e4e68fyahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:13px;"><div><div dir="ltr" data-setdir="false">Hi Mikael Nousiainen,<div><div data-test-id="message-view-body" class="ydp421b10e7I_ZkbNhI ydp421b10e7D_FY ydp421b10e7W_6D6F" style="width: 928.35px;"><div class="ydp421b10e7jb_0 ydp421b10e7X_6MGW ydp421b10e7N_6Fd5"></div></div><div class="ydp421b10e7H_7jIs ydp421b10e7D_F ydp421b10e7ab_C ydp421b10e7Q_69H5 ydp421b10e7E_36RhU" data-test-id="toolbar-hover-area"><div class="ydp421b10e7D_F ydp421b10e7W_6D6F ydp421b10e7r_BN ydp421b10e7gl_C" data-test-id="card-toolbar" style="width: 956.35px;"></div></div></div></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"> There is no buffer to queue the data received, any cranky timestamp can block the sink and pipeline can go to dead lock. Introducing the quue may help. Default its 1-second buffering adjust it if you need.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><span><span style="font-family: Helvetica Neue, Helvetica, Arial, sans-serif;">gst-launch-1.0 udpsrc address=127.0.0.1 port=22101 reuse=FALSE caps="application/x-rtp" ! rtpopusdepay ! opusdec ! audio/x-raw, rate=48000, channels=1, format=S16LE ! audioconvert ! audioresample ! queue leaky=2 ! pulsesink device="alsa_output.usb-BurrBrown_from_Texas_Instruments_USB_AUDIO_CODEC-00.analog-stereo"</span></span><br></div><div class="ydp12e4e68fsignature"><div><br></div><div><br></div><div>Sent from Yahoo Mail. <a href="https://yho.com/148vdq" rel="nofollow" target="_blank">Get the app</a></div></div></div> <div><br></div><div><br></div> </div><div id="yahoo_quoted_8782277768" class="yahoo_quoted"> <div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;"> <div> On Sunday, 26 February, 2023 at 04:05:39 am GMT-6, Mikael Nousiainen via gstreamer-devel <gstreamer-devel@lists.freedesktop.org> wrote: </div> <div><br></div> <div><br></div> <div><div dir="ltr">I managed to get some additional info on where the pipeline MIGHT stop working.<br clear="none"><br clear="none">I got this stack trace from gst-launch-1.0 (version 1.18.4) using GDB:<br clear="none"><br clear="none">#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x1fcc0dc) at ../sysdeps/nptl/futex-internal.h:186<br clear="none">#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x13, cond=0x1fcc0b0) at pthread_cond_wait.c:508<br clear="none">#2 __pthread_cond_wait (cond=0x1fcc0b0, mutex=0x13) at pthread_cond_wait.c:638<br clear="none">#3 0xb5048864 in pa_threaded_mainloop_wait () at /lib/arm-linux-gnueabihf/libpulse.so.0<br clear="none">#4 0xb5071738 in gst_pulseringbuffer_commit (buf=0x80, sample=0x906e7b5c, data=0x220d938 "", in_samples=<optimized out>, out_samples=<optimized out>, accum=0xb00fdc08) at ../ext/pulse/pulsesink.c:1585<br clear="none">#5 0xb53578d4 in () at /lib/arm-linux-gnueabihf/libgstaudio-1.0.so.0<br clear="none"><br clear="none">Looks like the pipeline might be stuck in pulsesink.c:1585 -- I checked the stack trace multiple times over a couple of minutes, exiting GDB in between to give GStreamer time to proceed. I got the same result every time.<br clear="none"><br clear="none">I can see that the line 1585 is this one in GitHub (for version 1.18.4):<br clear="none"><br clear="none"><a shape="rect" href="https://github.com/GStreamer/gst-plugins-good/blob/1.18.4/ext/pulse/pulsesink.c#L1585" target="_blank">https://github.com/GStreamer/gst-plugins-good/blob/1.18.4/ext/pulse/pulsesink.c#L1585</a><br clear="none"><br clear="none">So it may be that it's the pulsesink that gets stuck (instead of udpsrc).<br clear="none"><br clear="none">Based on the code it looks like there is no space to write to the PulseAudio sink...?<br clear="none"><br clear="none"> /* we can't write segsize bytes, wait a bit */<br clear="none"> GST_LOG_OBJECT (psink, "waiting for free space");<br clear="none"><br clear="none"> pa_threaded_mainloop_wait (mainloop); // <- this is line 1585<br clear="none"><br clear="none">What could this mean? The sound card is still there, PulseAudio is working (audio can be recorded all the time) and restarting the pipeline will fix playback too!<br clear="none"><br clear="none">For reference, here are the complete stack traces from gst-launch-1.0:<br clear="none"><br clear="none">Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".<br clear="none">__GI___poll (timeout=-1, nfds=2, fds=0x2251ee0) at ../sysdeps/unix/sysv/linux/poll.c:29<br clear="none">29 ../sysdeps/unix/sysv/linux/poll.c: No such file or directory.<br clear="none">(gdb) info threads<br clear="none"> Id Target Id Frame <br clear="none">* 1 Thread 0xb6f17e00 (LWP 1334) "gst-launch-1.0" __GI___poll (timeout=-1, nfds=2, fds=0x2251ee0) at ../sysdeps/unix/sysv/linux/poll.c:29<br clear="none"> 2 Thread 0xb4a17440 (LWP 1389) "threaded-ml" __GI___poll (timeout=1499, nfds=2, fds=0xb0103d40) at ../sysdeps/unix/sysv/linux/poll.c:29<br clear="none"> 3 Thread 0xb00ff440 (LWP 1392) "udpsrc0:src" futex_wait_cancelable (private=0, expected=0, futex_word=0x1fcc0dc) at ../sysdeps/nptl/futex-internal.h:186<br clear="none"> 4 Thread 0xaf8fe440 (LWP 1393) "gmain" __GI___poll (timeout=-1, nfds=1, fds=0x22480f0) at ../sysdeps/unix/sysv/linux/poll.c:29<br clear="none">(gdb) bt<br clear="none">#0 __GI___poll (timeout=-1, nfds=2, fds=0x2251ee0) at ../sysdeps/unix/sysv/linux/poll.c:29<br clear="none">#1 __GI___poll (fds=0x2251ee0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:26<br clear="none">#2 0xb6cba988 in () at /lib/arm-linux-gnueabihf/libglib-2.0.so.0<br clear="none">(gdb) t 2<br clear="none">[Switching to thread 2 (Thread 0xb4a17440 (LWP 1389))]<br clear="none">#0 __GI___poll (timeout=1499, nfds=2, fds=0xb0103d40) at ../sysdeps/unix/sysv/linux/poll.c:29<br clear="none">29 in ../sysdeps/unix/sysv/linux/poll.c<br clear="none">(gdb) bt<br clear="none">#0 __GI___poll (timeout=1499, nfds=2, fds=0xb0103d40) at ../sysdeps/unix/sysv/linux/poll.c:29<br clear="none">#1 __GI___poll (fds=0xb0103d40, nfds=2, timeout=1499) at ../sysdeps/unix/sysv/linux/poll.c:26<br clear="none">#2 0xb5047f90 in () at /lib/arm-linux-gnueabihf/libpulse.so.0<br clear="none">(gdb) t 3<br clear="none">[Switching to thread 3 (Thread 0xb00ff440 (LWP 1392))]<br clear="none">#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x1fcc0dc) at ../sysdeps/nptl/futex-internal.h:186<br clear="none">186 ../sysdeps/nptl/futex-internal.h: No such file or directory.<br clear="none">(gdb) bt<br clear="none">#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x1fcc0dc) at ../sysdeps/nptl/futex-internal.h:186<br clear="none">#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x13, cond=0x1fcc0b0) at pthread_cond_wait.c:508<br clear="none">#2 __pthread_cond_wait (cond=0x1fcc0b0, mutex=0x13) at pthread_cond_wait.c:638<br clear="none">#3 0xb5048864 in pa_threaded_mainloop_wait () at /lib/arm-linux-gnueabihf/libpulse.so.0<br clear="none">#4 0xb5071738 in gst_pulseringbuffer_commit (buf=0x80, sample=0x906e7b5c, data=0x220d938 "", in_samples=<optimized out>, out_samples=<optimized out>, accum=0xb00fdc08) at ../ext/pulse/pulsesink.c:1585<br clear="none">#5 0xb53578d4 in () at /lib/arm-linux-gnueabihf/libgstaudio-1.0.so.0<br clear="none">(gdb) t 4<br clear="none">[Switching to thread 4 (Thread 0xaf8fe440 (LWP 1393))]<br clear="none">#0 __GI___poll (timeout=-1, nfds=1, fds=0x22480f0) at ../sysdeps/unix/sysv/linux/poll.c:29<br clear="none">29 ../sysdeps/unix/sysv/linux/poll.c: No such file or directory.<br clear="none">(gdb) bt<br clear="none">#0 __GI___poll (timeout=-1, nfds=1, fds=0x22480f0) at ../sysdeps/unix/sysv/linux/poll.c:29<br clear="none">#1 __GI___poll (fds=0x22480f0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:26<br clear="none">#2 0xb6cba988 in () at /lib/arm-linux-gnueabihf/libglib-2.0.so.0<br clear="none"><br clear="none">-Mikael<br clear="none"><div class="yqt8772668381" id="yqtfd84477"><br clear="none">On Fri, Feb 24, 2023, at 11:28, Mikael Nousiainen via gstreamer-devel wrote:<br clear="none">>> 1.18 is fairly old, it would be great if you could try a newer version<br clear="none">>> such as 1.22.x.<br clear="none">><br clear="none">> I'm using Debian Bullseye and so far haven't found anything newer.<br clear="none">> Are there any places with newer binaries for Debian distributions?<br clear="none">> Compiling all of GStreamer sounds like a big task... :)<br clear="none">><br clear="none">> That said, the same bug has been precent for at least 4 years,<br clear="none">> but for some reason this has become more prominent recently<br clear="none">> with the pipeline stopping more often.<br clear="none">><br clear="none">>> <br clear="none">>> One thing you could do after you noticed it stopped receiving packets<br clear="none">>> is to attach a debugger such as gdb to gst-launch-1.0 via gdb -p `pidof<br clear="none">>> gst-launch-1.0` and then get a stack trace of all threads to see where<br clear="none">>> they're stuck and what they're waiting on.<br clear="none">><br clear="none">> I'll try this... once I get another failure!<br clear="none">><br clear="none">>> You could write a little application in C or python that creates a<br clear="none">>> pipeline using gst_parse_launch() and starts it up.<br clear="none">>> <br clear="none">>> With an application you can enable the ring buffer logger<br clear="none">>> (gst_debug_add_ring_buffer_logger) to continuously log into memory.<br clear="none">>> <br clear="none">>> You could then use a watchdog element in your pipeline to detect when<br clear="none">>> data stops flowing, at which point you can then grab the last X MB or<br clear="none">>> seconds of debug log from the ringbuffer and write it somewhere.<br clear="none">><br clear="none">> I'm afraid this gets a bit complicated.<br clear="none">><br clear="none">>> <br clear="none">>> Cheers<br clear="none">>> ?Tim<br clear="none">><br clear="none">> -Mikael<br clear="none">><br clear="none">> On Wed, Feb 22, 2023, at 11:54, Mikael Nousiainen wrote:<br clear="none">>> I've got a working pipeline that streams RTP/Opus audio from Janus <br clear="none">>> Gateway to a PulseAudio sink.<br clear="none">>><br clear="none">>> When the pipeline works, everything is fine, even in terms of latency.<br clear="none">>><br clear="none">>> The pipeline command is:<br clear="none">>><br clear="none">>> gst-launch-1.0 udpsrc address=127.0.0.1 port=22101 reuse=FALSE <br clear="none">>> caps="application/x-rtp" ! rtpopusdepay ! opusdec ! audio/x-raw, <br clear="none">>> rate=48000, channels=1, format=S16LE ! audioconvert ! audioresample ! <br clear="none">>> pulsesink <br clear="none">>> device="alsa_output.usb-BurrBrown_from_Texas_Instruments_USB_AUDIO_CODEC-00.analog-stereo"<br clear="none">>><br clear="none">>> However, the pipeline stops working randomly. The time frame could be <br clear="none">>> 10 minutes or 2 weeks,<br clear="none">>> but eventually gst-launch-1.0 process stops receiving UDP packets and I <br clear="none">>> see them piling up<br clear="none">>> in the Recv-Q (receive queue) when checking netstat. I have also <br clear="none">>> confirmed that the UDP packets<br clear="none">>> keep on coming from Janus Gateway even if the pipeline fails to receive <br clear="none">>> them (that's why they end up in the queue).<br clear="none">>><br clear="none">>> See netstat output when the pipeline is NOT working (see high Recv-Q value):<br clear="none">>><br clear="none">>> Active Internet connections (servers and established)<br clear="none">>> Proto Recv-Q Send-Q Local Address Foreign Address <br clear="none">>> State User Inode PID/Program name <br clear="none">>> tcp 0 0 172.20.0.2:49376 172.20.0.1:4713 <br clear="none">>> ESTABLISHED 1000 4026621 650/gst-launch-1.0 <br clear="none">>> udp 173888 0 127.0.0.1:22101 0.0.0.0:* <br clear="none">>> 1000 4023945 650/gst-launch-1.0 <br clear="none">>><br clear="none">>> See netstat output below when the pipeline is working fine:<br clear="none">>><br clear="none">>> Proto Recv-Q Send-Q Local Address Foreign Address <br clear="none">>> State User Inode PID/Program name <br clear="none">>> tcp 0 0 172.20.0.2:49436 172.20.0.1:4713 <br clear="none">>> ESTABLISHED 1000 9268042 1710/gst-launch-1.0 <br clear="none">>> udp 0 0 127.0.0.1:22101 0.0.0.0:* <br clear="none">>> 1000 9270388 1710/gst-launch-1.0 <br clear="none">>><br clear="none">>> The RTP/Opus UDP packet stream from Janus Gateway is not constant, <br clear="none">>> meaning that it may stop if there is no audio to be played. However, <br clear="none">>> I've noticed that the pipeline issue does not correspond with packets <br clear="none">>> not being present, as sometimes the failure happens right after <br clear="none">>> restarting the pipeline while audio is available.<br clear="none">>><br clear="none">>> Restarting the pipeline process always fixes the issue.<br clear="none">>><br clear="none">>> I've checked that the gst-launch-1.0 process is still somehow alive, <br clear="none">>> because it does send some sort of "keep-alive" packets to PulseAudio <br clear="none">>> (via UNIX socket) even after stopping to process UDP packets.<br clear="none">>><br clear="none">>> I've also got a similar pipeline working in the opposite direction, <br clear="none">>> streaming audio from PulseAudio source and sending it as an Opus/RTP <br clear="none">>> audio stream to Janus Gateway and that pipeline NEVER has any issues.<br clear="none">>><br clear="none">>> There is no output from gst-launch-1.0 process when the UDP reception <br clear="none">>> stops and I have not noticed any kernel messages at those times either. <br clear="none">>> It is difficult for me to enable very verbose logging in <br clear="none">>> gst-launch-1.0, as the issue might take weeks to show up and there's <br clear="none">>> really no easy way to find space for the verbose logs.<br clear="none">>><br clear="none">>> Would you have any ideas what could stop UDP packet reception in <br clear="none">>> GStreamer udpsrc? I've attempted to alter the "reuse" parameter, but it <br clear="none">>> doesn't seem to have any effect.<br clear="none">>><br clear="none">>> Could it still be that some other process can "steal" the UDP stream <br clear="none">>> from gst-launch-1.0? I'm out of clues here :)<br clear="none">>><br clear="none">>> GStreamer version details:<br clear="none">>><br clear="none">>> # gst-launch-1.0 --version<br clear="none">>> gst-launch-1.0 version 1.18.4<br clear="none">>> GStreamer 1.18.4<br clear="none">>> <a shape="rect" href="http://packages.qa.debian.org/gstreamer1.0" target="_blank">http://packages.qa.debian.org/gstreamer1.0</a><br clear="none">>><br clear="none">>> Thanks,<br clear="none">>> Mikael Nousiainen<br clear="none"></div></div></div> </div> </div></body></html>