Synchronized virtual cameras

[Andrew Silby]

Hi Jonas,
Does the below help? It can be run on the same machine in different terminals or on different machines if you change the IP addresses. The clocks on each session will sync and stay in sync.

Terminal 1:
gst-launch-1.0 rtpbin buffer-mode=4 ntp-sync=true name=rtpbin rtp-profile=avpf rtcp-sync-send-time=FALSE videotestsrc is-live=true ! video/x-raw,width=640,height=480,framerate=20/1 ! queue  !  clockoverlay ! queue ! videoconvert ! queue ! x264enc speed-preset=8  ! queue ! h264parse ! video/x-h264,profile=main,width=640,height=480,framerate=20/1  ! queue !  rtph264pay ! queue ! rtpbin.send_rtp_sink_0 rtpbin.send_rtp_src_0 ! udpsink host=127.0.0.1 port=10000 sync=false async=false rtpbin.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=10001 sync=false async=false udpsrc port=10002 ! rtpbin.recv_rtcp_sink_0

Terminal 2:
gst-launch-1.0 rtpbin name=rtpbin buffer-mode=4 ntp-sync=true rtp-profile=avpf rtcp-sync-send-time=FALSE videotestsrc is-live=true ! video/x-raw,width=640,height=480,framerate=20/1 ! queue  !  clockoverlay ! queue ! videoconvert ! queue ! x264enc speed-preset=2  ! queue ! h264parse ! video/x-h264,profile=main,width=640,height=480,framerate=20/1  ! queue !  rtph264pay ! queue ! rtpbin.send_rtp_sink_0 rtpbin.send_rtp_src_0 ! udpsink host=127.0.0.1 port=10003 sync=false async=false rtpbin.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=10004 sync=false async=false udpsrc port=10005 ! rtpbin.recv_rtcp_sink_0

Terminal 3:
gst-launch-1.0 rtpbin buffer-mode=4 max-rtcp-rtp-time-diff=5000 ntp-sync=true name=rtpbin2 latency=4000 rtcp-sync-send-time=FALSE rtp-profile=avpf compositor  name=convert  sink_0::xpos=0 sink_0::ypos=0 sink_0::zorder=0 sink_1::xpos=100 sink_1::ypos=100 sink_1::zorder=1  ! queue ! videoscale ! video/x-raw,width=640,height=480 ! queue ! videoconvert ! queue ! autovideosink udpsrc caps="application/x-rtp,media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=96" port=10000 ! rtpbin2.recv_rtp_sink_0 rtpbin2. ! queue ! rtph264depay ! queue ! avdec_h264 ! video/x-raw,width=640,height=480,framerate=20/1 ! queue ! videoconvert ! queue ! convert.sink_0 udpsrc port=10001 ! rtpbin2.recv_rtcp_sink_0 rtpbin2.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=10002 sync=false async=false   udpsrc caps="application/x-rtp,media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=96" port=10003 ! rtpbin2.recv_rtp_sink_1 rtpbin2. ! queue ! rtph264depay !  queue ! avdec_h264 ! video/x-raw,width=640,height=480,framerate=20/1 ! queue ! videoconvert !  queue ! convert.sink_1 udpsrc port=10004 ! rtpbin2.recv_rtcp_sink_1 rtpbin2.send_rtcp_src_1 ! udpsink host=127.0.0.1 port=10005 sync=false async=false


I hope this helps.
Andy

-----Original Message-----
From: gstreamer-devel <gstreamer-devel-bounces at lists.freedesktop.org> On Behalf Of Jonas Danielsson via gstreamer-devel
Sent: 08 March 2023 15:39
To: gstreamer-devel at lists.freedesktop.org
Cc: Jonas Danielsson <jonas.danielsson at spiideo.com>
Subject: Synchronized virtual cameras

Hi hello hackers!

We are trying to great some developer tools to help us with our current project!
In our system we receive live (RTP) feeds from two (or more) cameras and stitch them together and perform some fancy operations on the result.

It would really help us two be able to do this on pre-defined inputs!

To be more clear we would like to create a pipeline that mimics two cameras sending a live feed of the same thing, from different angles.
To be even more clear, a pipeline that sends two different video files over RTP/RTCP, as if they occurred live. And the rest of our system being non the wiser.

Is there a clear way of doing this? We have done some attempts, syncing to NTP clock, but sooner or later we seem to always have some kind of drift between the two virtual cameras.

Is this a really hard problem or is it in fact an easy problem but we are missing something?
Does anyone have any pointers for us for how we could implement this "Virtual Camera System".

Thanks for your time and for your work on GStreamer!

Regards
Jonas