Synchronized virtual cameras

[Jonas Danielsson]

Thank you Andrew!

I do wonder though if this will do the trick, it is quite similar to
what we are trying today.
What we need is the content of the two files we stream to line up as
if they where watching the same thing live.
So the first video frame from the first file should align, as well as
possible, and be synced with the first frame from the second file and
keep that sync wthout drift.
There is no `is-live` property on a filesrc as far as I know, so it is
a bit tricky for us how to do this ...

... we need to throttle the sending of file content I guess?
... and we need someway to make sure that the content of the files
line up and is synced, and that sync is kept.

If we "just" make sure the pipeline is synced to NTP clock and the RTP
bin uses capture time, is that really enough?
What happens when the files are read and sent with a slight difference
in time? How to we correlate the capture times?

Do I make sense? I hardly do to myself :)

All the best
Jonas

On Wed, 8 Mar 2023 at 17:12, Andrew Silby <asilby at yahoo.com> wrote:
>
> Hi Jonas,
> Does the below help? It can be run on the same machine in different terminals or on different machines if you change the IP addresses. The clocks on each session will sync and stay in sync.
>
> Terminal 1:
> gst-launch-1.0 rtpbin buffer-mode=4 ntp-sync=true name=rtpbin rtp-profile=avpf rtcp-sync-send-time=FALSE videotestsrc is-live=true ! video/x-raw,width=640,height=480,framerate=20/1 ! queue  !  clockoverlay ! queue ! videoconvert ! queue ! x264enc speed-preset=8  ! queue ! h264parse ! video/x-h264,profile=main,width=640,height=480,framerate=20/1  ! queue !  rtph264pay ! queue ! rtpbin.send_rtp_sink_0 rtpbin.send_rtp_src_0 ! udpsink host=127.0.0.1 port=10000 sync=false async=false rtpbin.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=10001 sync=false async=false udpsrc port=10002 ! rtpbin.recv_rtcp_sink_0
>
> Terminal 2:
> gst-launch-1.0 rtpbin name=rtpbin buffer-mode=4 ntp-sync=true rtp-profile=avpf rtcp-sync-send-time=FALSE videotestsrc is-live=true ! video/x-raw,width=640,height=480,framerate=20/1 ! queue  !  clockoverlay ! queue ! videoconvert ! queue ! x264enc speed-preset=2  ! queue ! h264parse ! video/x-h264,profile=main,width=640,height=480,framerate=20/1  ! queue !  rtph264pay ! queue ! rtpbin.send_rtp_sink_0 rtpbin.send_rtp_src_0 ! udpsink host=127.0.0.1 port=10003 sync=false async=false rtpbin.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=10004 sync=false async=false udpsrc port=10005 ! rtpbin.recv_rtcp_sink_0
>
> Terminal 3:
> gst-launch-1.0 rtpbin buffer-mode=4 max-rtcp-rtp-time-diff=5000 ntp-sync=true name=rtpbin2 latency=4000 rtcp-sync-send-time=FALSE rtp-profile=avpf compositor  name=convert  sink_0::xpos=0 sink_0::ypos=0 sink_0::zorder=0 sink_1::xpos=100 sink_1::ypos=100 sink_1::zorder=1  ! queue ! videoscale ! video/x-raw,width=640,height=480 ! queue ! videoconvert ! queue ! autovideosink udpsrc caps="application/x-rtp,media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=96" port=10000 ! rtpbin2.recv_rtp_sink_0 rtpbin2. ! queue ! rtph264depay ! queue ! avdec_h264 ! video/x-raw,width=640,height=480,framerate=20/1 ! queue ! videoconvert ! queue ! convert.sink_0 udpsrc port=10001 ! rtpbin2.recv_rtcp_sink_0 rtpbin2.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=10002 sync=false async=false   udpsrc caps="application/x-rtp,media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=96" port=10003 ! rtpbin2.recv_rtp_sink_1 rtpbin2. ! queue ! rtph264depay !  queue ! avdec_h264 ! video/x-raw,width=640,height=480,framerate=20/1 ! queue ! videoconvert !  queue ! convert.sink_1 udpsrc port=10004 ! rtpbin2.recv_rtcp_sink_1 rtpbin2.send_rtcp_src_1 ! udpsink host=127.0.0.1 port=10005 sync=false async=false
>
>
> I hope this helps.
> Andy
>
> -----Original Message-----
> From: gstreamer-devel <gstreamer-devel-bounces at lists.freedesktop.org> On Behalf Of Jonas Danielsson via gstreamer-devel
> Sent: 08 March 2023 15:39
> To: gstreamer-devel at lists.freedesktop.org
> Cc: Jonas Danielsson <jonas.danielsson at spiideo.com>
> Subject: Synchronized virtual cameras
>
> Hi hello hackers!
>
> We are trying to great some developer tools to help us with our current project!
> In our system we receive live (RTP) feeds from two (or more) cameras and stitch them together and perform some fancy operations on the result.
>
> It would really help us two be able to do this on pre-defined inputs!
>
> To be more clear we would like to create a pipeline that mimics two cameras sending a live feed of the same thing, from different angles.
> To be even more clear, a pipeline that sends two different video files over RTP/RTCP, as if they occurred live. And the rest of our system being non the wiser.
>
> Is there a clear way of doing this? We have done some attempts, syncing to NTP clock, but sooner or later we seem to always have some kind of drift between the two virtual cameras.
>
> Is this a really hard problem or is it in fact an easy problem but we are missing something?
> Does anyone have any pointers for us for how we could implement this "Virtual Camera System".
>
> Thanks for your time and for your work on GStreamer!
>
> Regards
> Jonas
>