high latency livestreaming question (h264/aac, rtp rtcp), playback in gstreamer-0.10 a/v doesn't sync a/v

[Ralf Stoye]

Hi,

I have some problems playing an stream with high latency in gstreamer. 

here a short version of what i try to do:

goal: livestream with highquality settings in x264, to an intermediant server in the internet, restream for different platforms.
current problem: audio / video async when receiving this with gstreamer

server: decklink ->  x264 -> big queue -> x264/voaacenc ->  payloader -> rtpbin(rtp&rtcp)   
test client: sdp-file in vlc or QT  Result: no problem a/v syncs after about a second 
gst client:   udpsrc -> jitterbuffer (2x for  rtp and rtcp) -> [big] queue -> depay  -> decode -> xvimagesink / alsasink    Result: A/V stays async (1-2 seconds)


and now the detailed version:
i try do stream live video & audio from a decklink card as an  h264/aac encoded rtp stream, and have problems playing this stream in gstreamer
no problem in VLC or Quicktime(7) Player:

using  a gstrtpbin with a high latency setting (5s) and apropriate big queues, i can use x264 with higher quality settings than "zerolatency" like you see in the following pipeline:

# sender pipeline
gst-launch-0.10 -vvv -m gstrtpbin latency=5000 name=rtpbin \
    decklinksrc mode=2 connection=4 audio-input=3 name=decklink0 \
    decklink0.audiosrc ! audioconvert ! "audio/x-raw-int,rate=48000,channels=2" ! queue max-size-buffers=40000 max-size-time=3000000000 max-size-bytes=0 ! voaacenc bitrate=128000 ! rtpmp4apay ! rtpbin.send_rtp_sink_0 \
        rtpbin.send_rtp_src_0 ! udpsink host=10.44.1.21 port=5008 name=udpaudio \
        rtpbin.send_rtcp_src_0 ! udpsink host=10.44.1.21 port=5009 sync=false async=false name=udpaudioc \
    decklink0.videosrc  ! ffmpegcolorspace ! queue ! \
	x264enc  bitrate=700 tune=film speed-preset=medium ! \
	  queue! rtph264pay ! rtpbin.send_rtp_sink_1 \
	rtpbin.send_rtp_src_1 ! udpsink host=10.44.1.21 port=5004 name=udpvid \
	rtpbin.send_rtcp_src_1 ! udpsink host=10.44.1.21 port=5005 sync=false async=false name=udpvidc

if i open the following sdp file in vlc or quicktime player the rtcp packets make the stream lip sync after about one or two seconds:
# sdp for client
v=0
o=- 1208520720 2590316915 IN IP4 10.44.1.21
c=IN IP4 10.44.1.21
s=ESP H264+AAC STREAM
m=video 5004 RTP/AVP 96
a=rtpmap:96 H264/90000
a=fmtp:96 media=video; clock-rate=90000; encoding-name=H264; sprop-parameter-sets=Z0LAHtkAtBJsCSIAAAMAAu5rKAAeLFyQAA==,aMuMsg==
a=control:trackID=1
m=audio 5008 RTP/AVP 96
a=rtpmap:96 MP4A-LATM/48000/2
a=fmtp:96 media=audio; clock-rate=48000; encoding-name=MP4A-LATM; cpresent=0; config=40002320; payload=96
a=control:trackID=2

Now i would like to stream like this to an intermediate server in the internet, which buffers the media and restreams it (with less  relative latency between audio and video, or in other formats).
So i first tried to *play* it locally with the following pipeline (yes, i adjusted ips of sending pipeline accordingly and copied all a/v caps from server to client):

gst-launch-0.10 -vvv -m --gst-debug gstrtpjitterbuffer:0 udpsrc caps="application/x-rtp, media=(string)audio, clock-rate=(int)48000, encoding-name=(string)MP4A-LATM, cpresent=(string)0, config=(string)40002320, ssrc=(uint)3270306329, payload=(int)96, clock-base=(uint)1916405448, seqnum-base=(uint)64596" port=5008 ! gstrtpjitterbuffer name=bufaudio latency=5000 ! queue max-size-buffers=40000 max-size-time=3000000000 max-size-bytes=0 ! rtpmp4adepay  ! queue ! ffdec_aac ! alsasink sync=false \
 udpsrc caps="application/x-rtcp" port=5009 !  bufaudio.sink_rtcp \
 udpsrc caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, sprop-parameter-sets=(string)\"Z0LAHtkAtBJsCSIAAAMAAu5rKAAeLFyQAA\\=\\=\\,aMuMsg\\=\\=\", ssrc=(uint)3913183102, payload=(int)96, clock-base=(uint)1743606014, seqnum-base=(uint)58420" port=5004 ! gstrtpjitterbuffer name=bufvideo latency=5000 ! queue max-size-buffers=40000 max-size-time=3000000000 max-size-bytes=0 ! rtph264depay ! ffdec_h264 ! xvimagesink sync=false display="127.0.0.1:0" \
 udpsrc caps="application/x-rtcp" port=5005 !  bufvideo.sink_rtcp

By turning on debugging for the jitterbuffers, i noticed they dropped the rtcp sync caused by the fact that the buffers where "too far ahead" (see gstrtpjitterbuffer.c line 2037) 
So i changed some x264 parameters to reach a latency smaller that clockrate (rc-lookahead,bframes), the messages were gone now, fine.
Unfortenately audio and video are  still off by about one second.


Any ideas on this?
Thank you for reading this long post

Ralf