Syncing two live streams

[Andrew Silby]

Hi Scott,

Try something like this:

Have a look at the examples here:
https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good/html/gst-plugins-good-plugins-rtpbin.html

Or try something like this:

Computer 1
gst-launch-1.0 rtpbin name=rtpbin rtp-profile=avpf rtcp-sync-send-time=false videotestsrc is-live=true ! video/x-raw,width=1280,height=720,framerate=20/1 ! queue  !  clockoverlay ! videoconvert ! vp8enc ! video/x-vp8,width=1280,height=720,framerate=20/1  ! queue !  rtpvp8pay ! rtpbin.send_rtp_sink_0 rtpbin.send_rtp_src_0 ! udpsink host=127.0.0.1 port=10000 rtpbin.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=10001 sync=false async=false udpsrc port=10002 ! rtpbin.recv_rtcp_sink_0

Computer 2
gst-launch-1.0 rtpbin name=rtpbin rtp-profile=avpf rtcp-sync-send-time=false videotestsrc is-live=true ! video/x-raw,width=1280,height=720,framerate=20/1 ! queue  !  clockoverlay ! videoconvert ! vp8enc ! video/x-vp8,width=1280,height=720,framerate=20/1  ! queue !  rtpvp8pay ! rtpbin.send_rtp_sink_0 rtpbin.send_rtp_src_0 ! udpsink host=127.0.0.1 port=10003 rtpbin.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=10004 sync=false async=false udpsrc port=10005 ! rtpbin.recv_rtcp_sink_0

Computer 3
gst-launch-1.0 rtpbin  name=rtpbin2 rtp-profile=avpf compositor  name=convert  sink_0::xpos=0 sink_0::ypos=0 sink_0::zorder=0 sink_1::xpos=640 sink_1::ypos=360 sink_1::zorder=1  ! videoscale ! video/x-raw,width=1280,height=720,framerate=20/1 ! videoconvert ! queue max-size-buffers=0 max-size-bytes=0 max-size-time=0 ! autovideosink udpsrc caps="application/x-rtp,media=(string)video, clock-rate=(int)90000, encoding-name=(string)VP8, payload=96" port=10000 ! rtpbin2.recv_rtp_sink_0 rtpbin2. ! rtpvp8depay ! vp8dec ! videorate ! queue  max-size-buffers=0 max-size-bytes=0 max-size-time=0 ! video/x-raw,width=1280,height=720,framerate=20/1 ! videoconvert ! convert.sink_0 udpsrc port=10001 ! rtpbin2.recv_rtcp_sink_0 rtpbin2.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=10002 sync=false async=false   udpsrc caps="application/x-rtp,media=(string)video, clock-rate=(int)90000, encoding-name=(string)VP8, payload=96" port=10003 ! rtpbin2.recv_rtp_sink_1 rtpbin2. ! rtpvp8depay !  vp8dec ! videorate ! queue  max-size-buffers=0 max-size-bytes=0 max-size-time=0 ! video/x-raw,width=1280,height=720,framerate=20/1 ! videoconvert ! convert.sink_1 udpsrc port=10004 ! rtpbin2.recv_rtcp_sink_1 rtpbin2.send_rtcp_src_1 ! udpsink host=127.0.0.1 port=10005 sync=false async=false

You can add in audio as well in a similar way. I do exactly what you're describing below at the rtpbin framework is excellent.


-----Original Message-----
From: gstreamer-devel [mailto:gstreamer-devel-bounces at lists.freedesktop.org] On Behalf Of vbplaya
Sent: 11 June 2018 22:01
To: gstreamer-devel at lists.freedesktop.org
Subject: Syncing two live streams

Can someone point me the right direction to where I can sync up a live video and audio stream?

I know it sound simple but here is my issue: 

- We have 2 computers streaming to a single computer across multiple networks (which can be up to hundreds of miles away).
- All three computers have their system clocks synchronized using NTP
- Video computer gathers video and streams to the Display computer
- Audio computer gathers audio and also streams to the Display computer

There is an application which accepts the audio stream.  This application does two things (plays the audio over the speakers and sends network delay information to my application).  I am not privileged to the method which they stream the audio.
  
My application displays the video and two other tasks (which I haven't been able to figure out how to do yet).
- I need to be able to determine the network delay on the video stream (ideally, it would be great to have a timestamp on the video stream from the Video computer which is related to that system clock so I can compare that timestamp to my own system clock).
- I also need to delay the video display to allow it to be synced up with the audio.

Everything I have found assumes that either the audio and video are being streamed from the same computer, or that the audio stream is being done by gstreamer so I could use some sync function.  I am not privileged to the actual audio stream.  I am only given the amount of time the audio was delayed getting there (network delay).

So intermittently, I am given a number as the network delay for the audio
(example: 250 ms).  I need to be able to determine my own network delay for the video (which I don't know how to do yet).  Then I need to compare to see if the audio delay is more than the video network delay.  Say the video is 100ms ... then I would need to delay the video display by 150ms (which I also don't know how to do).  

ANY HELP is appreciated.  I am trying to pick up where someone else has left off in this design so it hasn't been easy for me to figure this out and move forward.  Also being done in Python ... which further limits the information I have been able to find. Thanks.  

Scott




--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
_______________________________________________
gstreamer-devel mailing list
gstreamer-devel at lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel