<!DOCTYPE html><html><head><title></title><style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div>Hi,<br></div><div><br></div><div>Thanks for the suggestion and possible explanation for the deadlock.<br></div><div><br></div><div>However, the requirement here is to play back audio in real time (think of it as a voice call), so I cannot add long delays, preferably none at all!<br></div><div><br></div><div>Losing a packet in case they arrive out of order is fine -- but is there any way to just skip timestamp/out-of-order errors instead of a deadlock?<br></div><div><br></div><div>-Mikael</div><div><br></div><div>On Tue, Mar 7, 2023, at 23:00, vinod kesti wrote:<br></div><blockquote type="cite" id="qt" style=""><div class="qt-ydp12e4e68fyahoo-style-wrap" style="font-family:"Helvetica Neue", Helvetica, Arial, sans-serif;font-size:13px;"><div><div dir="ltr"><div>Hi Mikael Nousiainen,<br></div><div><div class="qt-ydp421b10e7I_ZkbNhI qt-ydp421b10e7D_FY qt-ydp421b10e7W_6D6F" style="width:928.35px;"><div class="qt-ydp421b10e7jb_0 qt-ydp421b10e7X_6MGW qt-ydp421b10e7N_6Fd5"><br></div></div><div class="qt-ydp421b10e7H_7jIs qt-ydp421b10e7D_F qt-ydp421b10e7ab_C qt-ydp421b10e7Q_69H5 qt-ydp421b10e7E_36RhU"><div class="qt-ydp421b10e7D_F qt-ydp421b10e7W_6D6F qt-ydp421b10e7r_BN qt-ydp421b10e7gl_C" style="width:956.35px;"><br></div></div></div></div><div dir="ltr"><br></div><div dir="ltr"> There is no buffer to queue the data received, any cranky timestamp can block the sink and pipeline can go to dead lock. Introducing the quue may help. Default its 1-second buffering adjust it if you need.<br></div><div dir="ltr"><br></div><div dir="ltr"><span><span class="font" style="font-family:"Helvetica Neue", Helvetica, Arial, sans-serif;">gst-launch-1.0 udpsrc address=127.0.0.1 port=22101 reuse=FALSE caps="application/x-rtp" ! rtpopusdepay ! opusdec ! audio/x-raw, rate=48000, channels=1, format=S16LE ! audioconvert ! audioresample ! queue leaky=2 ! pulsesink device="alsa_output.usb-BurrBrown_from_Texas_Instruments_USB_AUDIO_CODEC-00.analog-stereo"</span></span><br></div><div class="qt-ydp12e4e68fsignature"><div><br></div><div><br></div><div>Sent from Yahoo Mail. <a href="https://yho.com/148vdq" rel="nofollow" target="_blank">Get the app</a><br></div></div></div><div><br></div><div><br></div></div><div id="qt-yahoo_quoted_8782277768" class="qt-yahoo_quoted"><div style="font-family:"Helvetica Neue", Helvetica, Arial, sans-serif;font-size:13px;color:rgb(38, 40, 42);"><div>On Sunday, 26 February, 2023 at 04:05:39 am GMT-6, Mikael Nousiainen via gstreamer-devel <gstreamer-devel@lists.freedesktop.org> wrote:<br></div><div><br></div><div><br></div><div><div dir="ltr"><div>I managed to get some additional info on where the pipeline MIGHT stop working.<br></div><div><br></div><div>I got this stack trace from gst-launch-1.0 (version 1.18.4) using GDB:<br></div><div><br></div><div>#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x1fcc0dc) at ../sysdeps/nptl/futex-internal.h:186<br></div><div>#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x13, cond=0x1fcc0b0) at pthread_cond_wait.c:508<br></div><div>#2 __pthread_cond_wait (cond=0x1fcc0b0, mutex=0x13) at pthread_cond_wait.c:638<br></div><div>#3 0xb5048864 in pa_threaded_mainloop_wait () at /lib/arm-linux-gnueabihf/libpulse.so.0<br></div><div>#4 0xb5071738 in gst_pulseringbuffer_commit (buf=0x80, sample=0x906e7b5c, data=0x220d938 "", in_samples=<optimized out>, out_samples=<optimized out>, accum=0xb00fdc08) at ../ext/pulse/pulsesink.c:1585<br></div><div>#5 0xb53578d4 in () at /lib/arm-linux-gnueabihf/libgstaudio-1.0.so.0<br></div><div><br></div><div>Looks like the pipeline might be stuck in pulsesink.c:1585 -- I checked the stack trace multiple times over a couple of minutes, exiting GDB in between to give GStreamer time to proceed. I got the same result every time.<br></div><div><br></div><div>I can see that the line 1585 is this one in GitHub (for version 1.18.4):<br></div><div><br></div><div><a shape="rect" href="https://github.com/GStreamer/gst-plugins-good/blob/1.18.4/ext/pulse/pulsesink.c#L1585" target="_blank">https://github.com/GStreamer/gst-plugins-good/blob/1.18.4/ext/pulse/pulsesink.c#L1585</a><br></div><div><br></div><div>So it may be that it's the pulsesink that gets stuck (instead of udpsrc).<br></div><div><br></div><div>Based on the code it looks like there is no space to write to the PulseAudio sink...?<br></div><div><br></div><div> /* we can't write segsize bytes, wait a bit */<br></div><div> GST_LOG_OBJECT (psink, "waiting for free space");<br></div><div><br></div><div> pa_threaded_mainloop_wait (mainloop); // <- this is line 1585<br></div><div><br></div><div>What could this mean? The sound card is still there, PulseAudio is working (audio can be recorded all the time) and restarting the pipeline will fix playback too!<br></div><div><br></div><div>For reference, here are the complete stack traces from gst-launch-1.0:<br></div><div><br></div><div>Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".<br></div><div>__GI___poll (timeout=-1, nfds=2, fds=0x2251ee0) at ../sysdeps/unix/sysv/linux/poll.c:29<br></div><div>29 ../sysdeps/unix/sysv/linux/poll.c: No such file or directory.<br></div><div>(gdb) info threads<br></div><div> Id Target Id Frame <br></div><div>* 1 Thread 0xb6f17e00 (LWP 1334) "gst-launch-1.0" __GI___poll (timeout=-1, nfds=2, fds=0x2251ee0) at ../sysdeps/unix/sysv/linux/poll.c:29<br></div><div> 2 Thread 0xb4a17440 (LWP 1389) "threaded-ml" __GI___poll (timeout=1499, nfds=2, fds=0xb0103d40) at ../sysdeps/unix/sysv/linux/poll.c:29<br></div><div> 3 Thread 0xb00ff440 (LWP 1392) "udpsrc0:src" futex_wait_cancelable (private=0, expected=0, futex_word=0x1fcc0dc) at ../sysdeps/nptl/futex-internal.h:186<br></div><div> 4 Thread 0xaf8fe440 (LWP 1393) "gmain" __GI___poll (timeout=-1, nfds=1, fds=0x22480f0) at ../sysdeps/unix/sysv/linux/poll.c:29<br></div><div>(gdb) bt<br></div><div>#0 __GI___poll (timeout=-1, nfds=2, fds=0x2251ee0) at ../sysdeps/unix/sysv/linux/poll.c:29<br></div><div>#1 __GI___poll (fds=0x2251ee0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:26<br></div><div>#2 0xb6cba988 in () at /lib/arm-linux-gnueabihf/libglib-2.0.so.0<br></div><div>(gdb) t 2<br></div><div>[Switching to thread 2 (Thread 0xb4a17440 (LWP 1389))]<br></div><div>#0 __GI___poll (timeout=1499, nfds=2, fds=0xb0103d40) at ../sysdeps/unix/sysv/linux/poll.c:29<br></div><div>29 in ../sysdeps/unix/sysv/linux/poll.c<br></div><div>(gdb) bt<br></div><div>#0 __GI___poll (timeout=1499, nfds=2, fds=0xb0103d40) at ../sysdeps/unix/sysv/linux/poll.c:29<br></div><div>#1 __GI___poll (fds=0xb0103d40, nfds=2, timeout=1499) at ../sysdeps/unix/sysv/linux/poll.c:26<br></div><div>#2 0xb5047f90 in () at /lib/arm-linux-gnueabihf/libpulse.so.0<br></div><div>(gdb) t 3<br></div><div>[Switching to thread 3 (Thread 0xb00ff440 (LWP 1392))]<br></div><div>#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x1fcc0dc) at ../sysdeps/nptl/futex-internal.h:186<br></div><div>186 ../sysdeps/nptl/futex-internal.h: No such file or directory.<br></div><div>(gdb) bt<br></div><div>#0 futex_wait_cancelable (private=0, expected=0, futex_word=0x1fcc0dc) at ../sysdeps/nptl/futex-internal.h:186<br></div><div>#1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x13, cond=0x1fcc0b0) at pthread_cond_wait.c:508<br></div><div>#2 __pthread_cond_wait (cond=0x1fcc0b0, mutex=0x13) at pthread_cond_wait.c:638<br></div><div>#3 0xb5048864 in pa_threaded_mainloop_wait () at /lib/arm-linux-gnueabihf/libpulse.so.0<br></div><div>#4 0xb5071738 in gst_pulseringbuffer_commit (buf=0x80, sample=0x906e7b5c, data=0x220d938 "", in_samples=<optimized out>, out_samples=<optimized out>, accum=0xb00fdc08) at ../ext/pulse/pulsesink.c:1585<br></div><div>#5 0xb53578d4 in () at /lib/arm-linux-gnueabihf/libgstaudio-1.0.so.0<br></div><div>(gdb) t 4<br></div><div>[Switching to thread 4 (Thread 0xaf8fe440 (LWP 1393))]<br></div><div>#0 __GI___poll (timeout=-1, nfds=1, fds=0x22480f0) at ../sysdeps/unix/sysv/linux/poll.c:29<br></div><div>29 ../sysdeps/unix/sysv/linux/poll.c: No such file or directory.<br></div><div>(gdb) bt<br></div><div>#0 __GI___poll (timeout=-1, nfds=1, fds=0x22480f0) at ../sysdeps/unix/sysv/linux/poll.c:29<br></div><div>#1 __GI___poll (fds=0x22480f0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:26<br></div><div>#2 0xb6cba988 in () at /lib/arm-linux-gnueabihf/libglib-2.0.so.0<br></div><div><br></div><div>-Mikael<br></div><div class="qt-yqt8772668381" id="qt-yqtfd84477"><div><br></div><div>On Fri, Feb 24, 2023, at 11:28, Mikael Nousiainen via gstreamer-devel wrote:<br></div><div>>> 1.18 is fairly old, it would be great if you could try a newer version<br></div><div>>> such as 1.22.x.<br></div><div>><br></div><div>> I'm using Debian Bullseye and so far haven't found anything newer.<br></div><div>> Are there any places with newer binaries for Debian distributions?<br></div><div>> Compiling all of GStreamer sounds like a big task... :)<br></div><div>><br></div><div>> That said, the same bug has been precent for at least 4 years,<br></div><div>> but for some reason this has become more prominent recently<br></div><div>> with the pipeline stopping more often.<br></div><div>><br></div><div>>> <br></div><div>>> One thing you could do after you noticed it stopped receiving packets<br></div><div>>> is to attach a debugger such as gdb to gst-launch-1.0 via gdb -p `pidof<br></div><div>>> gst-launch-1.0` and then get a stack trace of all threads to see where<br></div><div>>> they're stuck and what they're waiting on.<br></div><div>><br></div><div>> I'll try this... once I get another failure!<br></div><div>><br></div><div>>> You could write a little application in C or python that creates a<br></div><div>>> pipeline using gst_parse_launch() and starts it up.<br></div><div>>> <br></div><div>>> With an application you can enable the ring buffer logger<br></div><div>>> (gst_debug_add_ring_buffer_logger) to continuously log into memory.<br></div><div>>> <br></div><div>>> You could then use a watchdog element in your pipeline to detect when<br></div><div>>> data stops flowing, at which point you can then grab the last X MB or<br></div><div>>> seconds of debug log from the ringbuffer and write it somewhere.<br></div><div>><br></div><div>> I'm afraid this gets a bit complicated.<br></div><div>><br></div><div>>> <br></div><div>>> Cheers<br></div><div>>> ?Tim<br></div><div>><br></div><div>> -Mikael<br></div><div>><br></div><div>> On Wed, Feb 22, 2023, at 11:54, Mikael Nousiainen wrote:<br></div><div>>> I've got a working pipeline that streams RTP/Opus audio from Janus <br></div><div>>> Gateway to a PulseAudio sink.<br></div><div>>><br></div><div>>> When the pipeline works, everything is fine, even in terms of latency.<br></div><div>>><br></div><div>>> The pipeline command is:<br></div><div>>><br></div><div>>> gst-launch-1.0 udpsrc address=127.0.0.1 port=22101 reuse=FALSE <br></div><div>>> caps="application/x-rtp" ! rtpopusdepay ! opusdec ! audio/x-raw, <br></div><div>>> rate=48000, channels=1, format=S16LE ! audioconvert ! audioresample ! <br></div><div>>> pulsesink <br></div><div>>> device="alsa_output.usb-BurrBrown_from_Texas_Instruments_USB_AUDIO_CODEC-00.analog-stereo"<br></div><div>>><br></div><div>>> However, the pipeline stops working randomly. The time frame could be <br></div><div>>> 10 minutes or 2 weeks,<br></div><div>>> but eventually gst-launch-1.0 process stops receiving UDP packets and I <br></div><div>>> see them piling up<br></div><div>>> in the Recv-Q (receive queue) when checking netstat. I have also <br></div><div>>> confirmed that the UDP packets<br></div><div>>> keep on coming from Janus Gateway even if the pipeline fails to receive <br></div><div>>> them (that's why they end up in the queue).<br></div><div>>><br></div><div>>> See netstat output when the pipeline is NOT working (see high Recv-Q value):<br></div><div>>><br></div><div>>> Active Internet connections (servers and established)<br></div><div>>> Proto Recv-Q Send-Q Local Address Foreign Address <br></div><div>>> State User Inode PID/Program name <br></div><div>>> tcp 0 0 172.20.0.2:49376 172.20.0.1:4713 <br></div><div>>> ESTABLISHED 1000 4026621 650/gst-launch-1.0 <br></div><div>>> udp 173888 0 127.0.0.1:22101 0.0.0.0:* <br></div><div>>> 1000 4023945 650/gst-launch-1.0 <br></div><div>>><br></div><div>>> See netstat output below when the pipeline is working fine:<br></div><div>>><br></div><div>>> Proto Recv-Q Send-Q Local Address Foreign Address <br></div><div>>> State User Inode PID/Program name <br></div><div>>> tcp 0 0 172.20.0.2:49436 172.20.0.1:4713 <br></div><div>>> ESTABLISHED 1000 9268042 1710/gst-launch-1.0 <br></div><div>>> udp 0 0 127.0.0.1:22101 0.0.0.0:* <br></div><div>>> 1000 9270388 1710/gst-launch-1.0 <br></div><div>>><br></div><div>>> The RTP/Opus UDP packet stream from Janus Gateway is not constant, <br></div><div>>> meaning that it may stop if there is no audio to be played. However, <br></div><div>>> I've noticed that the pipeline issue does not correspond with packets <br></div><div>>> not being present, as sometimes the failure happens right after <br></div><div>>> restarting the pipeline while audio is available.<br></div><div>>><br></div><div>>> Restarting the pipeline process always fixes the issue.<br></div><div>>><br></div><div>>> I've checked that the gst-launch-1.0 process is still somehow alive, <br></div><div>>> because it does send some sort of "keep-alive" packets to PulseAudio <br></div><div>>> (via UNIX socket) even after stopping to process UDP packets.<br></div><div>>><br></div><div>>> I've also got a similar pipeline working in the opposite direction, <br></div><div>>> streaming audio from PulseAudio source and sending it as an Opus/RTP <br></div><div>>> audio stream to Janus Gateway and that pipeline NEVER has any issues.<br></div><div>>><br></div><div>>> There is no output from gst-launch-1.0 process when the UDP reception <br></div><div>>> stops and I have not noticed any kernel messages at those times either. <br></div><div>>> It is difficult for me to enable very verbose logging in <br></div><div>>> gst-launch-1.0, as the issue might take weeks to show up and there's <br></div><div>>> really no easy way to find space for the verbose logs.<br></div><div>>><br></div><div>>> Would you have any ideas what could stop UDP packet reception in <br></div><div>>> GStreamer udpsrc? I've attempted to alter the "reuse" parameter, but it <br></div><div>>> doesn't seem to have any effect.<br></div><div>>><br></div><div>>> Could it still be that some other process can "steal" the UDP stream <br></div><div>>> from gst-launch-1.0? I'm out of clues here :)<br></div><div>>><br></div><div>>> GStreamer version details:<br></div><div>>><br></div><div>>> # gst-launch-1.0 --version<br></div><div>>> gst-launch-1.0 version 1.18.4<br></div><div>>> GStreamer 1.18.4<br></div><div>>> <a shape="rect" href="http://packages.qa.debian.org/gstreamer1.0" target="_blank">http://packages.qa.debian.org/gstreamer1.0</a><br></div><div>>><br></div><div>>> Thanks,<br></div><div>>> Mikael Nousiainen<br></div></div></div></div></div></div></blockquote></body></html>