Gstreamer RTSP & splitmux issues

[Will McElderry]

On 12/04/2023 11:22, W McElderry via gstreamer-devel wrote:
> Hi All,
>
>
> First up:  Thank you to all those who have contributed to the project 
> - it's very useful and appreciated!
>
>
> I've got a complex set of issues when using gstreamer through 
> pygobject and gst-launch-1.0 to capture some live RTSP camera streams 
> to video files.  From what I've read, that doesn't sound very novel!
>
> What I've done with a splitmuxsink/splitmuxsrc to try and mitigate the 
> issue may be more interesting as I think that has revealed a bug 
> somewhere in the code (though I'd like to lay it out for someone who 
> knows more to comment!)
>
>
> Before I dive in: is this type of discussion or is there somewhere 
> better?
>
>
> Thanks!
>
>
> Will.
>

Hi All,


I've written this while waiting for a response - and then just decided 
to send it.  It is large, but I've tried to make it easy to read and 
address questions.


*NB:* I have discovered the following using an old gstreamer version 
(*1.14.5*) - I cannot upgrade the system, but I'm running the pipeline 
again now through docker with 22.04 as a base (*1.20.3*).  I haven't 
looked at building gstreamer from source.



Here's the story in order:

I'm trying to reliably record data from a group of security cameras 
using RTSP/RTP H.264, but it's failing in some cameras/situations.

One set of cameras I'm attempting to capture from is misbehaving - I 
believe it is reporting that it is changing it's framerate, even though 
it has been manually configured to be 15 FPS.  That causes the MP4 mux 
to get upset in this pipeline:

WARNING: from element /GstPipeline:pipeline0/GstMP4Mux:mp4mux0: Can't 
change input format at runtime.
Additional debug info:
gstqtmux.c(5757): gst_qt_mux_video_sink_set_caps (): 
/GstPipeline:pipeline0/GstMP4Mux:mp4mux0: pad video_0 refused 
renegotiation to video/x-h264, stream-format=(string)avc, 
alignment=(string)au, codec_data=(buffer)<trimmed>, level=(string)3.1, 
profile=(string)high, pixel-aspect-ratio=(fraction)1/1, width=(int)1280, 
height=(int)720, framerate=(fraction)*7/1*, 
interlace-mode=(string)progressive, chroma-format=(string)4:2:0, 
bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true

sometimes it chooses other framerates too.

*Question 1*:  Is there a mux which will handle this better and just 
keep right on recording?



*Question (set) 2*: Possible workarounds?

When this happens, it seems like the pipeline halts: no more data is 
written to file, the MOOV atom is not written and the process doesn't 
terminate until a further sigINT is sent, where it terminates without 
wring the MOOV atom.

Consequence is the recording is not useful.

I think I've seen similar issues with the python equivalent 
implementation - the EOS is generated, but there is no way to terminate 
gracefully anymore.   Is that the case, or am I missing something?  Does 
that constitute a bug or a feature?

The primary work-a-round I came up with was to use --gst-fatal-warnings, 
but that doesn't terminate the process whie using the -e flag too.  
Perhaps that constitutes a bug, even if the MOOV atom not being written 
may not be?

*Update:* since writing the above, I've discovered mp4mux settings for 
reserved-duration-remaining and reserved-moov-update-period which seem 
to ensure I end up with a valid file.  I'm interested by 
moov-recovery-file, don't know how to make it work and note it is 
'experimental'.  Any comments appreciated!




(Written before the update above) To minimise the impact of the issue, I 
decided to use a tee and splitmuxsink so I'd have some valid data close 
to the point the issue occurred, or the whole file if no issue occurred 
as before:


For clarity pipeline described by:

     gst-launch-1.0 --gst-fatal-warnings -e rtspsrc "location=${URI}" ! \
         rtpjitterbuffer ! \
         queue ! \
         rtph264depay ! \
         h264parse ! \
         tee name=t ! \
         t. ! \
         splitmuxsink location=${FILE}.%05d max-size-time=$((60*1000000000)\
         t. ! \
         mp4mux ! \
         filesink "location=${FILE}" &

#aside:  I'm not sure of the impact of the rtpjitterbuffer - is this 
embedded within the rtspsrc, which seems to be a bin?


I tested the system and in time I saw a similar issue and attempted to 
use the split files to recover, then I discovered some situations (A,B 
and C below) I didn't expect that /may/ indicate something is wrong (or 
may be just my knowledge/use, but in case it's useful here we go)


A) Reassembling splits does not produce the same output as the 
monolithic file:

(NB: this situation persists when running with *1.20.3*, though input 
files were created with *1.14.5*)

Looking at the duration of the combined file and the original capture, 
it is clear these are not the same as the duration is shorter for the 
recombined file.

*Question 3:* Probably my limited understanding, but is it expected that 
reassembling a large h264 stream from a set of split files would yield a 
different output to storing the same h264 stream from an RTSP/RTP server 
directly?



B) Splitting the monolithic file does not recreate matching split files

(NB: this situation persists when running with *1.20.3*, though 
monolithic input file was created with *1.14.5*)

Taking the monolithic MP4 file instead of the rtspsrc and using the same 
command to repackage the encapsulated h264 stream through a splitmuxsink 
with the same parameters produces a different number of split files, so 
they must have different content.

As I'm not transcoding this is a bit of a surprise:  I thought the h264 
stream cannot be changed during this process and the information about 
splitting files would be deterministically derived from information 
embedded within the h264 stream, so I cannot see how this can happen.

*Question 4:* Probably my limited understanding, but is that expected?


C) Sometimes the split files cannot be successfully recombined

(NB: this situation persists when running with *1.20.3*, though input 
files were created with *1.14.5*)

Given that all split files are derived from one stream that seems to 
play normally, and the individual split files seem to 'play' normally 
(using splitmuxsrc location=${single_file}), I'm surprised that when 
using splitmuxsrc location=${multifile_glob} they can reliably produce 
errors (mp4mux: Buffer has no PTS.)

*Question 5:* Does that make sense in some way?


Any other tips to improve reliable recordings greatly appreciated!




I'll send through any more technical details I can if they're deemed 
useful (though I cannot send any actual data captured - best I can do is 
debug logs, carefully reviewed).


Thanks for your time!


Will.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20230414/72497b8a/attachment.htm>