appsrc blocks pipeline / how to handle unreliable data source

[Peter Biro]

Hi,

I have a pipeline where I want to mux together streaming video audio and
sensor readings / text data as a subtitle track to a video file. I
don't want to use a text overlay to burn the subtitle to the video, just to
have it as a separate track, which can be turned on or off on the client
side. Also want to embed this data into the video file directly,
therefore a separate srt file wont be a solution for me.

To achieve this I planned to use an "appsrc" element and push the text data
to it programmatically from a simple python app.

My current pipeline looks like this:
matroskamux name=output_queue_mux\
! filesink name=output_queue_filesink
location=/data/videos_staging/0_1683496385.mkv\
udpsrc multicast-group=0.0.0.0 auto-multicast=true port=56001
name=input_queue_video_udpsrc\
!
application/x-rtp,media=video,encoding-name=H264,payload=96,clock-rate=90000,packetization-mode=1,profile-level-id=424015,sprop-parameter-sets="Z0JAKJWgHgCJ+VA=,aM48gA==",src=1728323247,timestamp-offset=2499875162,seqnum-offset=11758,a-framerate=30\
! rtpjitterbuffer name=input_queue_video_rtpjitterbuffer ! rtph264depay
name=input_queue_video_rtpdepay\
! h264parse name=input_queue_video_parse ! queue name=blocking_video_queue\
! omxh264dec ! videoconvert ! omxh264enc bitrate=8000000 control-rate=2
insert-sps-pps=true\
! output_queue_mux.\
udpsrc multicast-group=0.0.0.0 auto-multicast=true port=51000
name=input_queue_audio_udpsrc\
!
application/x-rtp,media=audio,clock-rate=44100,encoding-name=L24,encoding-params=1,channels=1,payload=96,ssrc=687131883,timestamp-offset=3784732336,seqnum-offset=8272\
! rtpL24depay name=input_queue_audio_rtpL24depay\
! audioconvert name=input_queue_audio_audioconvert\
! input_queue_audio_adder.
audiotestsrc wave=silence name=input_queue_audio_audiotestsrc\
! audiomixer name=input_queue_audio_adder\
! audioresample name=input_queue_audio_audioresample ! voaacenc
name=input_queue_audio_voaacenc\
! queue name=blocking_audio_queue ! output_queue_mux.\
appsrc name=subtitle_appsrc format=GST_FORMAT_TIME
stream-type=GST_APP_STREAM_TYPE_SEEKABLE is-live=true do-timestamp=true !
text/x-raw !queue leaky=downstream ! output_queue_mux.

My problem with this is that sensor readings are arriving occasionally
(maybe no readings at all during recording) and as it does hold up the
whole pipeline as it is try to wait for data from all of the inputs.

I have some kinda similar situation with the audio stream, which is also
not reliable so I added a "audiotestsrc" to generate audio even when I
don't receive anything from the stream.

Do you have any suggestions on what could be a solution to my problem?

Is there any way to mark a source option in mux? Or is there any option to
achieve something similar like with the audio to always send some dummy
data with timestamps to unblock the pipeline?

Thank you!

Bests,
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20230507/b21e82f0/attachment.htm>