gstreamer inter-stream synchronization on separate machines

[Vasile Popescu]

Hi all,

Is it possible to synchronize audio with video on different hosts
(i.e. stream audio and video form a server and playback the audio on
one client/machine and video on other client/machine)?

Right now I am using a pipe to stream RTP/RTCP  video and audio data
to two different clients (running on two different machines) and I
want the video to be synchronized with audio. I've created the
pipelines (for server and clients) based on the example in here:
http://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/tests/examples/rtp/server-v4l2-H264-alsasrc-PCMA.sh

The pipes I'm using are:

on the server side:
gst-launch-1.0 --gst-debug=2 --verbose rtpbin name=rtpbin latency=0
filesrc location=../battleship.mp4 ! queue ! qtdemux name=demux
demux.video_0 ! queue ! rtph264pay mtu=1400 ! rtpbin.send_rtp_sink_0
rtpbin.send_rtp_src_0 ! udpsink port=5000 host=192.168.93.25
ts-offset=0 name=vrtpsink rtpbin.send_rtcp_src_0 ! udpsink port=5001
host=192.168.93.25 sync=false async=false name=vrtcpsink udpsrc
port=5005 name=vrtpsrc ! rtpbin.recv_rtcp_sink_0 demux.audio_0 ! queue
! rtpmp4apay mtu=1400 ! rtpbin.send_rtp_sink_1 rtpbin.send_rtp_src_1 !
udpsink port=5002 host=192.168.93.10 ts-offset=0 name=artpsink
rtpbin.send_rtcp_src_1 ! udpsink port=5003 host=192.168.93.10
sync=false async=false name=artcpsink udpsrc port=5007 name=artpsrc !
rtpbin.recv_rtcp_sink_1
gst-launch-1.0 --gst-debug=2 --verbose rtpbin name=rtpbin latency=0
filesrc location=../battleship.mp4 ! queue ! qtdemux name=demux
demux.video_0 ! queue ! rtph264pay mtu=1400 ! rtpbin.send_rtp_sink_0
rtpbin.send_rtp_src_0 ! udpsink port=5000 host=192.168.93.25
ts-offset=0 name=vrtpsink rtpbin.send_rtcp_src_0 ! udpsink port=5001
host=192.168.93.25 sync=false async=false name=vrtcpsink udpsrc
port=5005 name=vrtpsrc ! rtpbin.recv_rtcp_sink_0 demux.audio_0 ! queue
! rtpmp4apay mtu=1400 ! rtpbin.send_rtp_sink_1 rtpbin.send_rtp_src_1 !
udpsink port=5002 host=192.168.93.10 ts-offset=0 name=artpsink
rtpbin.send_rtcp_src_1 ! udpsink port=5003 host=192.168.93.10
sync=false async=false name=artcpsink udpsrc port=5007 name=artpsrc !
rtpbin.recv_rtcp_sink_1

on the audio client side:
gst-launch --gst-debug=2 --verbose gstrtpbin name=rtpbin latency=1000
udpsrc caps=application/x-rtp,media=(string)audio,clock-rate=(int)44100,encoding-name=(string)MP4A-LATM,cpresent=(string)0,config=(string)40002420,payload=(int)96
port=5002 ! rtpbin.recv_rtp_sink_0 rtpbin. ! rtpmp4adepay ! faad !
queue ! audioconvert ! audioresample ! pulsesink udpsrc port=5003 !
rtpbin.recv_rtcp_sink_0 rtpbin.send_rtcp_src_0 ! udpsink port=5007
host=192.168.93.10 sync=false async=false

on the video client side:
gst-launch --gst-debug=2 --verbose gstrtpbin name=rtpbin latency=4000
udpsrc caps=application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264,sprop-parameter-sets=(string)\"Z2QAKKzZQHgGewFqAgICgAAAAwCAAV+QB4wYyw\=\=\,aOvgjLIs\",payload=(int)96
port=5000 ! rtpbin.recv_rtp_sink_0 rtpbin. ! rtph264depay ! ffdec_h264
! ffmpegcolorspace ! autovideosink udpsrc port=5001 !
rtpbin.recv_rtcp_sink_0 rtpbin.send_rtcp_src_0 ! udpsink port=5005
host=192.168.93.10 sync=false async=false

Without too much load on the LAN the two clients appear to be
synchronized really well (lip-sync) .. but if I introduce some latency
to the packets sent from the server side (using this "tc qdisc add dev
eth0 root netem delay 500ms" on the server side) then the two clients
(which are running on different hosts) are going out of sync and never
come back to be synchronized.

Long story short, is it possible to do inter-stream synchronization on
different hosts? (each stream to play on its own host machine)? If
yes, how the pipe should look like?
What about multicast communication? Should the synchronization work
also over multicast? In the multicast communication the RTCP will be
only from the server to the clients (i.e. only RTCP SR packets), so is
that enough for inter-stream synchronization on multiple hosts?

I'm really looking forward to an answer so hopefully I will get this clarified.

Thank you,
Vasile.