Pipeline not cleans up after itself if it is started as a service

[Peter Biro]

Hi all!

We are developing a streaming application on Nvidia Jetson Nano platform, where we use GStreamer pipelines for part of the processing. We experienced a very strange behaviour when running the pipeline as a systemd service, when we try to stop the service the pipeline is stopped but it does not cleans up after itself which will result an error in the nvargus-daemon. We have multiple pipelines running simultaneously so a fail in the nvargus-daemon will cause errors in all of the pipelines.

Strange thing is that if we start the pipeline from a terminal session and hit CTRL-C or kill the process from and other session using 'kill -SIGINT <pid>' the pipeline cleans up after itself properly and I can start and stop it anytime without issues.

We have this pipeline:

gst-launch-1.0 -e\
 nvarguscamerasrc sensor-id="0" sensor-mode=0 gainrange="1 16" ispdigitalgainrange="1 1" name="camera_pipeline_overrides_0"\
    !  "video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1"\
    ! nvvidconv ! nvivafilter cuda-process=true customer-lib-name="libdcd_overlay_1080p.so" ! 'video/x-raw(memory:NVMM), format=(string)NV12'\
    ! nvvidconv ! nvv4l2vp8enc bitrate="8000000" control-rate=1 ! rtpvp8pay mtu=1400\
    ! udpsink auto-multicast=true clients="127.0.0.1:56000,127.0.0.1:56001"

But I was able to reproduce the issue with much more simple setup:

service definition:
$ cat /etc/systemd/system/dcd_camerasource_0.service
[Unit]
Description=Dual Channel Dashcam Camera Source
After=network.target syslog.target dcd.service dcd_gps.service dcd_radar.service nvargus-daemon.service
StartLimitIntervalSec=5

[Service]
Type=simple
Restart=always
RestartSec=5
User=dcd
Group=dcd
WorkingDirectory=/opt/dcd
ExecStart=/bin/bash /opt/dcd/bin/dcd_camerasource.sh 0
ExecStop=/bin/kill -s SIGINT -$MAINPID 
TimeoutStopSec=3

[Install]
WantedBy=multi-user.target

wrapper script for the pipeline:
$ cat /opt/dcd/bin/dcd_camerasource.sh
#!/bin/bash

gst-launch-1.0 nvarguscamerasrc sensor-id="$1" sensor-mode=0 ! fakesink

which can reproduce the issue. If I start the script itself I can start and stop it as many times I wish and the pipeline cleans up after itself. But if I use:
systemctl start dcd_camerasource_0
systemctl stop dcd_camerasource_0

Then I get similar errors:
Jan 28 16:07:30 dcd-398f9dee nvargus-daemon[4998]: === gst-launch-1.0[6785]: Connection closed (7F77AB11D0)=== gst-launch-1.0[6785]: WARNING: CameraProvider was not destroyed before client connection terminated.=== gst-launch-1.0[6785]:          The client may have abnormally terminated. Destroying CameraProvider...=== gst-launch-1.0[6785]: CameraProvider destroyed (0x7f70b5c4c0)=== gst-launch-1.0[6785]: WARNING: Cleaning up 1 outstanding requests...=== gst-launch-1.0[6785]: WARNING: Cleaning up 1 outstanding streams...SCF: Error InvalidState: 5 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 306 )
Jan 28 16:07:30 dcd-398f9dee nvargus-daemon[4998]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 50)
Jan 28 16:07:30 dcd-398f9dee nvargus-daemon[4998]: === gst-launch-1.0[6785]: WARNING: Cleaning up 1 outstanding stream settings...=== gst-launch-1.0[6785]: WARNING: Cleaning up 1 outstanding sessions...(NvCameraUtils) Error InvalidState: Mutex not initialized (/dvs/git/dirty/git-master_linux/camera/core_scf/src/services/gl/EGLStreamProducer.cpp:498) (in Mutex.cpp, function lock(), line 79)
Jan 28 16:07:30 dcd-398f9dee nvargus-daemon[4998]: SCF: Error BadParameter: Buffer is not pending (in src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 502)
Jan 28 16:07:30 dcd-398f9dee nvargus-daemon[4998]: (NvCameraUtils) Error InvalidState: Mutex has not been initialized (in Mutex.cpp, function unlock(), line 88)
Jan 28 16:07:30 dcd-398f9dee nvargus-daemon[4998]: SCF: Error BadParameter:  (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBuffer(), line 487)
Jan 28 16:07:30 dcd-398f9dee nvargus-daemon[4998]: SCF: Error BadParameter:  (propagating from src/components/CaptureContainerImpl.cpp, function returnBuffer(), line 447)
Jan 28 16:07:30 dcd-398f9dee nvargus-daemon[4998]: SCF: Error BadParameter:  (propagating from src/components/stages/BufferReturnStage.h, function doExecute(), line 43)
Jan 28 16:07:30 dcd-398f9dee nvargus-daemon[4998]: SCF: Error BadParameter: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 990)
Jan 28 16:07:30 dcd-398f9dee nvargus-daemon[4998]: (NvCameraUtils) Error InvalidState: Mutex not initialized (/dvs/git/dirty/git-master_linux/camera/core_scf/src/services/gl/EGLStreamProducer.cpp:498) (in Mutex.cpp, function lock(), line 79)
Jan 28 16:07:30 dcd-398f9dee nvargus-daemon[4998]: (NvCameraUtils) Error InvalidState: Mutex has not been initialized (in Mutex.cpp, function unlock(), line 88)

Do you have any idea what can be the difference between the two cases? Or what am I messing up when I starting it as a service?

Thank you for the help!

Bests,
Peter

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210803/b8779f35/attachment-0001.htm>