GStreamer multi pipeline time sync

[Peter Biro]

Hi all,

Im not sure if im at the right place to ask this, feel free to redirect me a more proper forum (I just started to familiarize with GStreamer so any feedback appreciated).

Im working on an application which consist of multiple pipelines:
Video source: capture video, do some simple operation on it (encode and overlaying) and passing the output to an appsink
Audio source: capture audio and forward it to an appsink
Streamer: sending the video (and maybe later audio as well) over as UDP stream or WebRTC (live video)
Buffer: a plain c++ class which stores the buffers for a configured time (it can be changed dynamically, and the audio and video buffer time can be different, so it should be possible that the video starts a few minutes before from the buffer before the audio)
Filewriter: this is a simple filesink, when its triggered I push the buffers to it's audio and video appsrc with the gst_app_src_push_buffer methond (at the moment I encode the video with VP8 to a webm file using a webmmux).

The pipelines are started at the same time (sequentially after each other), and pushing the buffers to the filewriter starts when a record request arrives.

VideoSource ----------------->    LiveStream
			|
			\
			   ------------>		  	    record trigger
						Buffer---------------- X ---------------->    FileWriter
AudioSource ----------------->


I have two kinds of issues:
I cannot make the audio and video sync in the filewriter
During playback the playtime counter not starts from 0 (I think it is somehow receiving the timestamp from the Video source, so when I start the video in VLC it start countig from eg.: 30 seconds which is the amount of time I waited until I triggered the record) and Im not able seek in the video file.

I tried to play around with following approaches:
the 'do-timestamp' and 'is-live' properties on the filewriter's receiving appsrc
reset the timestamps on the buffers (audio and video) with 'GST_BUFFER_PTS' to a sequentially value starting from zero and  increased with the duration
adjust the timestamps on the buffers (audio and video) with 'GST_BUFFER_PTS' with the time difference between the start times of the source and filewriter timelines
tried to sync all piplines by setting the same clock on them and use the same basetime on them

I think I miss some fundamental thing here. Can you give me some hints how should I properly do this time synchronisation?

Thanks a lot!

Bests,
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20201130/cd5331a2/attachment.htm>