GStreamer pipeline udpsrc randomly stops receiving UDP packets

[James]

> On 24 Feb 2023, at 5:28 pm, Mikael Nousiainen via gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
> 
>> 1.18 is fairly old, it would be great if you could try a newer version
>> such as 1.22.x.
> 
> I'm using Debian Bullseye and so far haven't found anything newer.
> Are there any places with newer binaries for Debian distributions?
> Compiling all of GStreamer sounds like a big task... :)
> 
> That said, the same bug has been precent for at least 4 years,
> but for some reason this has become more prominent recently
> with the pipeline stopping more often.
> 
>> 
>> One thing you could do after you noticed it stopped receiving packets
>> is to attach a debugger such as gdb to gst-launch-1.0 via gdb -p `pidof
>> gst-launch-1.0` and then get a stack trace of all threads to see where
>> they're stuck and what they're waiting on.
> 
> I'll try this... once I get another failure!
> 
>> You could write a little application in C or python that creates a
>> pipeline using gst_parse_launch() and starts it up.
>> 
>> With an application you can enable the ring buffer logger
>> (gst_debug_add_ring_buffer_logger) to continuously log into memory.
>> 
>> You could then use a watchdog element in your pipeline to detect when
>> data stops flowing, at which point you can then grab the last X MB or
>> seconds of debug log from the ringbuffer and write it somewhere.
> 
> I'm afraid this gets a bit complicated.
> 
>> 
>> Cheers
>> ?Tim
> 
> -Mikael
> 
> On Wed, Feb 22, 2023, at 11:54, Mikael Nousiainen wrote:
>> I've got a working pipeline that streams RTP/Opus audio from Janus 
>> Gateway to a PulseAudio sink.
>> 
>> When the pipeline works, everything is fine, even in terms of latency.
>> 
>> The pipeline command is:
>> 
>> gst-launch-1.0 udpsrc address=127.0.0.1 port=22101 reuse=FALSE 
>> caps="application/x-rtp" ! rtpopusdepay ! opusdec ! audio/x-raw, 
>> rate=48000, channels=1, format=S16LE ! audioconvert ! audioresample ! 
>> pulsesink 
>> device="alsa_output.usb-BurrBrown_from_Texas_Instruments_USB_AUDIO_CODEC-00.analog-stereo"
>> 
>> However, the pipeline stops working randomly. The time frame could be 
>> 10 minutes or 2 weeks,
>> but eventually gst-launch-1.0 process stops receiving UDP packets and I 
>> see them piling up
>> in the Recv-Q (receive queue) when checking netstat. I have also 
>> confirmed that the UDP packets
>> keep on coming from Janus Gateway even if the pipeline fails to receive 
>> them (that's why they end up in the queue).
>> 
>> See netstat output when the pipeline is NOT working (see high Recv-Q value):
>> 
>> Active Internet connections (servers and established)
>> Proto Recv-Q Send-Q Local Address           Foreign Address         
>> State       User       Inode      PID/Program name    
>> tcp        0      0 172.20.0.2:49376        172.20.0.1:4713         
>> ESTABLISHED 1000       4026621    650/gst-launch-1.0  
>> udp   173888      0 127.0.0.1:22101         0.0.0.0:*                   
>>        1000       4023945    650/gst-launch-1.0  
>> 
>> See netstat output below when the pipeline is working fine:
>> 
>> Proto Recv-Q Send-Q Local Address           Foreign Address         
>> State       User       Inode      PID/Program name    
>> tcp        0      0 172.20.0.2:49436        172.20.0.1:4713         
>> ESTABLISHED 1000       9268042    1710/gst-launch-1.0 
>> udp        0      0 127.0.0.1:22101         0.0.0.0:*                   
>>        1000       9270388    1710/gst-launch-1.0 
>> 
>> The RTP/Opus UDP packet stream from Janus Gateway is not constant, 
>> meaning that it may stop if there is no audio to be played. However, 
>> I've noticed that the pipeline issue does not correspond with packets 
>> not being present, as sometimes the failure happens right after 
>> restarting the pipeline while audio is available.
>> 
>> Restarting the pipeline process always fixes the issue.
>> 
>> I've checked that the gst-launch-1.0 process is still somehow alive, 
>> because it does send some sort of "keep-alive" packets to PulseAudio 
>> (via UNIX socket) even after stopping to process UDP packets.
>> 
>> I've also got a similar pipeline working in the opposite direction, 
>> streaming audio from PulseAudio source and sending it as an Opus/RTP 
>> audio stream to Janus Gateway and that pipeline NEVER has any issues.
>> 
>> There is no output from gst-launch-1.0 process when the UDP reception 
>> stops and I have not noticed any kernel messages at those times either. 
>> It is difficult for me to enable very verbose logging in 
>> gst-launch-1.0, as the issue might take weeks to show up and there's 
>> really no easy way to find space for the verbose logs.
>> 
>> Would you have any ideas what could stop UDP packet reception in 
>> GStreamer udpsrc? I've attempted to alter the "reuse" parameter, but it 
>> doesn't seem to have any effect.
>> 
>> Could it still be that some other process can "steal" the UDP stream 
>> from gst-launch-1.0? I'm out of clues here :)
>> 
>> GStreamer version details:
>> 
>> # gst-launch-1.0 --version
>> gst-launch-1.0 version 1.18.4
>> GStreamer 1.18.4
>> http://packages.qa.debian.org/gstreamer1.0

Actually I followed the wiki and did build from scratch. It was slow but painless.
The final bit to build an install not a test bed was findable if not obvious.

'Course you could change your repo. Try one of the rolling releases like tumbleweed or arch. Manjaro is an easy way to do arch. Gentoo folk: sorry I've not tried.
James