GStreamer pipeline udpsrc randomly stops receiving UDP packets

[Mikael Nousiainen]

> 1.18 is fairly old, it would be great if you could try a newer version
> such as 1.22.x.

I'm using Debian Bullseye and so far haven't found anything newer.
Are there any places with newer binaries for Debian distributions?
Compiling all of GStreamer sounds like a big task... :)

That said, the same bug has been precent for at least 4 years,
but for some reason this has become more prominent recently
with the pipeline stopping more often.

> 
> One thing you could do after you noticed it stopped receiving packets
> is to attach a debugger such as gdb to gst-launch-1.0 via gdb -p `pidof
> gst-launch-1.0` and then get a stack trace of all threads to see where
> they're stuck and what they're waiting on.

I'll try this... once I get another failure!

> You could write a little application in C or python that creates a
> pipeline using gst_parse_launch() and starts it up.
> 
> With an application you can enable the ring buffer logger
> (gst_debug_add_ring_buffer_logger) to continuously log into memory.
> 
> You could then use a watchdog element in your pipeline to detect when
> data stops flowing, at which point you can then grab the last X MB or
> seconds of debug log from the ringbuffer and write it somewhere.

I'm afraid this gets a bit complicated.

> 
> Cheers
> ?Tim

-Mikael

On Wed, Feb 22, 2023, at 11:54, Mikael Nousiainen wrote:
> I've got a working pipeline that streams RTP/Opus audio from Janus 
> Gateway to a PulseAudio sink.
>
> When the pipeline works, everything is fine, even in terms of latency.
>
> The pipeline command is:
>
> gst-launch-1.0 udpsrc address=127.0.0.1 port=22101 reuse=FALSE 
> caps="application/x-rtp" ! rtpopusdepay ! opusdec ! audio/x-raw, 
> rate=48000, channels=1, format=S16LE ! audioconvert ! audioresample ! 
> pulsesink 
> device="alsa_output.usb-BurrBrown_from_Texas_Instruments_USB_AUDIO_CODEC-00.analog-stereo"
>
> However, the pipeline stops working randomly. The time frame could be 
> 10 minutes or 2 weeks,
> but eventually gst-launch-1.0 process stops receiving UDP packets and I 
> see them piling up
> in the Recv-Q (receive queue) when checking netstat. I have also 
> confirmed that the UDP packets
> keep on coming from Janus Gateway even if the pipeline fails to receive 
> them (that's why they end up in the queue).
>
> See netstat output when the pipeline is NOT working (see high Recv-Q value):
>
> Active Internet connections (servers and established)
> Proto Recv-Q Send-Q Local Address           Foreign Address         
> State       User       Inode      PID/Program name    
> tcp        0      0 172.20.0.2:49376        172.20.0.1:4713         
> ESTABLISHED 1000       4026621    650/gst-launch-1.0  
> udp   173888      0 127.0.0.1:22101         0.0.0.0:*                   
>         1000       4023945    650/gst-launch-1.0  
>
> See netstat output below when the pipeline is working fine:
>
> Proto Recv-Q Send-Q Local Address           Foreign Address         
> State       User       Inode      PID/Program name    
> tcp        0      0 172.20.0.2:49436        172.20.0.1:4713         
> ESTABLISHED 1000       9268042    1710/gst-launch-1.0 
> udp        0      0 127.0.0.1:22101         0.0.0.0:*                   
>         1000       9270388    1710/gst-launch-1.0 
>
> The RTP/Opus UDP packet stream from Janus Gateway is not constant, 
> meaning that it may stop if there is no audio to be played. However, 
> I've noticed that the pipeline issue does not correspond with packets 
> not being present, as sometimes the failure happens right after 
> restarting the pipeline while audio is available.
>
> Restarting the pipeline process always fixes the issue.
>
> I've checked that the gst-launch-1.0 process is still somehow alive, 
> because it does send some sort of "keep-alive" packets to PulseAudio 
> (via UNIX socket) even after stopping to process UDP packets.
>
> I've also got a similar pipeline working in the opposite direction, 
> streaming audio from PulseAudio source and sending it as an Opus/RTP 
> audio stream to Janus Gateway and that pipeline NEVER has any issues.
>
> There is no output from gst-launch-1.0 process when the UDP reception 
> stops and I have not noticed any kernel messages at those times either. 
> It is difficult for me to enable very verbose logging in 
> gst-launch-1.0, as the issue might take weeks to show up and there's 
> really no easy way to find space for the verbose logs.
>
> Would you have any ideas what could stop UDP packet reception in 
> GStreamer udpsrc? I've attempted to alter the "reuse" parameter, but it 
> doesn't seem to have any effect.
>
> Could it still be that some other process can "steal" the UDP stream 
> from gst-launch-1.0? I'm out of clues here :)
>
> GStreamer version details:
>
> # gst-launch-1.0 --version
> gst-launch-1.0 version 1.18.4
> GStreamer 1.18.4
> http://packages.qa.debian.org/gstreamer1.0
>
> Thanks,
> Mikael Nousiainen