Synchronized virtual cameras

[asilby at yahoo.com]

Hi Jonas, I'm no expert in the non-live stuff. Maybe someone else can comment on that. But for live streams they sync and stay exactly in sync. I played around with this a few years ago. I have multiple cameras within a room, each connected to a separate raspberry pi or other similar device, all sending streams to a central computer which then brings them all together. It works perfectly.

In the test examples below I use a speed-preset of 8 and one of 2 to try and force the stream out of sync but they still sync up. Rtcp-sync-send-time should make sure it's the capture time that syncs.  The buufer-mode and max-rtcp-rtp-time-diff I seem to remember were fairly key to it working.

Andy

-----Original Message-----
From: Jonas Danielsson <jonas.danielsson at spiideo.com> 
Sent: 08 March 2023 16:40
To: Andrew Silby <asilby at yahoo.com>
Cc: Discussion of the development of and with GStreamer <gstreamer-devel at lists.freedesktop.org>
Subject: Re: Synchronized virtual cameras

Thank you Andrew!

I do wonder though if this will do the trick, it is quite similar to what we are trying today.
What we need is the content of the two files we stream to line up as if they where watching the same thing live.
So the first video frame from the first file should align, as well as possible, and be synced with the first frame from the second file and keep that sync wthout drift.
There is no `is-live` property on a filesrc as far as I know, so it is a bit tricky for us how to do this ...

... we need to throttle the sending of file content I guess?
... and we need someway to make sure that the content of the files line up and is synced, and that sync is kept.

If we "just" make sure the pipeline is synced to NTP clock and the RTP bin uses capture time, is that really enough?
What happens when the files are read and sent with a slight difference in time? How to we correlate the capture times?

Do I make sense? I hardly do to myself :)

All the best
Jonas

On Wed, 8 Mar 2023 at 17:12, Andrew Silby <asilby at yahoo.com> wrote:
>
> Hi Jonas,
> Does the below help? It can be run on the same machine in different terminals or on different machines if you change the IP addresses. The clocks on each session will sync and stay in sync.
>
> Terminal 1:
> gst-launch-1.0 rtpbin buffer-mode=4 ntp-sync=true name=rtpbin 
> rtp-profile=avpf rtcp-sync-send-time=FALSE videotestsrc is-live=true ! 
> video/x-raw,width=640,height=480,framerate=20/1 ! queue  !  
> clockoverlay ! queue ! videoconvert ! queue ! x264enc speed-preset=8  
> ! queue ! h264parse ! 
> video/x-h264,profile=main,width=640,height=480,framerate=20/1  ! queue 
> !  rtph264pay ! queue ! rtpbin.send_rtp_sink_0 rtpbin.send_rtp_src_0 ! 
> udpsink host=127.0.0.1 port=10000 sync=false async=false 
> rtpbin.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=10001 sync=false 
> async=false udpsrc port=10002 ! rtpbin.recv_rtcp_sink_0
>
> Terminal 2:
> gst-launch-1.0 rtpbin name=rtpbin buffer-mode=4 ntp-sync=true 
> rtp-profile=avpf rtcp-sync-send-time=FALSE videotestsrc is-live=true ! 
> video/x-raw,width=640,height=480,framerate=20/1 ! queue  !  
> clockoverlay ! queue ! videoconvert ! queue ! x264enc speed-preset=2  
> ! queue ! h264parse ! 
> video/x-h264,profile=main,width=640,height=480,framerate=20/1  ! queue 
> !  rtph264pay ! queue ! rtpbin.send_rtp_sink_0 rtpbin.send_rtp_src_0 ! 
> udpsink host=127.0.0.1 port=10003 sync=false async=false 
> rtpbin.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=10004 sync=false 
> async=false udpsrc port=10005 ! rtpbin.recv_rtcp_sink_0
>
> Terminal 3:
> gst-launch-1.0 rtpbin buffer-mode=4 max-rtcp-rtp-time-diff=5000 ntp-sync=true name=rtpbin2 latency=4000 rtcp-sync-send-time=FALSE rtp-profile=avpf compositor  name=convert  sink_0::xpos=0 sink_0::ypos=0 sink_0::zorder=0 sink_1::xpos=100 sink_1::ypos=100 sink_1::zorder=1  ! queue ! videoscale ! video/x-raw,width=640,height=480 ! queue ! videoconvert ! queue ! autovideosink udpsrc caps="application/x-rtp,media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=96" port=10000 ! rtpbin2.recv_rtp_sink_0 rtpbin2. ! queue ! rtph264depay ! queue ! avdec_h264 ! video/x-raw,width=640,height=480,framerate=20/1 ! queue ! videoconvert ! queue ! convert.sink_0 udpsrc port=10001 ! rtpbin2.recv_rtcp_sink_0 rtpbin2.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=10002 sync=false async=false   udpsrc caps="application/x-rtp,media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=96" port=10003 ! rtpbin2.recv_rtp_sink_1 rtpbin2. ! queue ! rtph264depay !  queue ! avdec_h264 ! video/x-raw,width=640,height=480,framerate=20/1 ! queue ! videoconvert !  queue ! convert.sink_1 udpsrc port=10004 ! rtpbin2.recv_rtcp_sink_1 rtpbin2.send_rtcp_src_1 ! udpsink host=127.0.0.1 port=10005 sync=false async=false
>
>
> I hope this helps.
> Andy
>
> -----Original Message-----
> From: gstreamer-devel <gstreamer-devel-bounces at lists.freedesktop.org> 
> On Behalf Of Jonas Danielsson via gstreamer-devel
> Sent: 08 March 2023 15:39
> To: gstreamer-devel at lists.freedesktop.org
> Cc: Jonas Danielsson <jonas.danielsson at spiideo.com>
> Subject: Synchronized virtual cameras
>
> Hi hello hackers!
>
> We are trying to great some developer tools to help us with our current project!
> In our system we receive live (RTP) feeds from two (or more) cameras and stitch them together and perform some fancy operations on the result.
>
> It would really help us two be able to do this on pre-defined inputs!
>
> To be more clear we would like to create a pipeline that mimics two cameras sending a live feed of the same thing, from different angles.
> To be even more clear, a pipeline that sends two different video files over RTP/RTCP, as if they occurred live. And the rest of our system being non the wiser.
>
> Is there a clear way of doing this? We have done some attempts, syncing to NTP clock, but sooner or later we seem to always have some kind of drift between the two virtual cameras.
>
> Is this a really hard problem or is it in fact an easy problem but we are missing something?
> Does anyone have any pointers for us for how we could implement this "Virtual Camera System".
>
> Thanks for your time and for your work on GStreamer!
>
> Regards
> Jonas
>