Gstreamer timestamps with start/stop recording

[Terry Barnaby]

I have been tearing my hair out for a while with this, nothing i try 
appears to do what I need!

The project is for an instrument that will display live camera video and 
record sections of interest under operator control. There are some 
complexities such as video overlays before display and H264 encoding, 
taking picture snapshots etc. Most of this is working but the ability to 
get a timestamp contiguous h264/mp4 video file.

When the video.mp4 file is played back by most players they show a fixed 
video frame for the entire time the record is off between record-off and 
record-on states due to the gstreamer timestamps incrementing while the 
record is off. I understand this would be what is normally wanted, but 
not in my case.

I have tried many approaches, to no avail, they all have some issues. I 
am testing under Fedora35 with the standard gstreamer 1.20.0, although 
the real system uses an embedded NXP IMX8mp. As a basic idea of my 
current approach which appears close to what I need (reduced test 
pipeline) I have something like the following.

This uses a special beamvalve plugin module. The beamvalve is a plugin 
based on the standard "valve" but modifies the buffer PTS time like the 
following when frames are not dropped:

static GstClockTime    time;
...
GST_BUFFER_PTS(buffer) = time;
time += GST_BUFFER_DURATION(buffer);

The pipeline is like:compositor name=c sink_1::alpha=1.0 ! videoconvert 
! identity drop-allocation=true ! tee name=t

videotestsrc pattern=smpte ! 
video/x-raw,width=640,height=480,framerate=5/1 ! c.sink_0
appsrc name="appsrc" ! imagefreeze ! videoconvert ! c.sink_1
t. ! queue ! glimagesink name=videoSink
t. ! queue ! fakesink enable-last-sample=true name=picturestream"
t. ! queue ! beamvalve name=recordValve drop=true ! vaapih264enc 
name=recordEncoder ! h264parse ! mp4mux fragment-duration=10 ! filesink 
append=true location="video.mp4" async=false

This pipeline is created in software and the recordValve is set to drop 
or not the video frames to disable/enable recording to the file.


This almost works (it does on a much simpler pipeline) but it has the 
issues:

1.When record is activated the GUI stops displaying new video frames for 
a few seconds and then starts again rather than keeps on going.

2. When record is de-activated the GUI stops displaying new video frames 
for a few seconds and then starts again and printf's inside beamvalve 
show no packets being processed while this is happening.

3. When I play the video with something like xine, it appears to show 
the video going slow then normal rate then slow again.

So questions:

1. Is there a better approach to doing this (having a time contiguous 
set of frames in the recorded ht64/mp4 file while display of camera 
video continues) ?

2. Any idea what may be going wrong with my approach ?

Cheers

Terry