Understanding timestamps in a complex pipeline

[Florian Echtler]

Hello everyone,

first of all, thanks to everyone who's been working on Gstreamer, it's made a
lot of things possible that would have taken ages otherwise :-)

I have built some fairly complex pipelines recently, which work well enough, but
for tuning some aspects, I'd like to understand queues and timestamping better.
Some of you may already have seen my SO posts, but a lot of things tend to get
lost in the noise there.

So for reference, here's my sender pipeline which grabs two video and one audio
stream, encodes them as H.264/Opus, wraps that into MP2TS and sends it over UDP
(taken from a wrapper shell script):

queue="queue max-size-time=200000000 leaky=downstream"

videoenc1="videoconvert ! $queue ! x264enc noise-reduction=10000
tune=zerolatency byte-stream=true threads=2 key-int-max=15"
videoenc2="videoconvert ! $queue ! vaapih264enc bitrate=2000 keyframe-period=10"
audioenc0="$queue ! opusenc bitrate=8000"

cmd="videotestsrc do-timestamp=true is-live=true !
video/x-raw,width=1280,height=720,framerate=10/1 ! $videoenc1 ! mux. \
  v4l2src  do-timestamp=true device=/dev/video0  !
video/x-raw,width=960,height=720,framerate=10/1  ! $videoenc2 ! mux. \
  pulsesrc do-timestamp=true                     !
audio/x-raw,channels=1,rate=16000                ! $audioenc0 ! mux. \
  mpegtsmux name=mux ! tee name=fork ! rtpmp2tpay ! udpsink host=$1 port=5000"

And here's the corresponding receiver pipeline:

udpsrc port=5000 ! application/x-rtp, media=video, clock-rate=90000,
encoding-name=MP2T ! rtpjitterbuffer ! rtpmp2tdepay ! tsparse ! tsdemux name=mux \
	mux. ! h264parse ! queue leaky=downstream ! avdec_h264 ! videoconvert !
fpsdisplaysink \
	mux. ! h264parse ! queue leaky=downstream ! avdec_h264 ! videoconvert !
fpsdisplaysink \
	mux. ! opusparse ! queue leaky=downstream ! opusdec ! autoaudiosink

Now, finally, for my questions:

1. I've realized only today that none of the sources set do-timestamp by
default, and I fail to understand why. I've had persistent synchronization
problems, and since I've set do-timestamp=true on all sources, everything seems
to be fine now. Why isn't this the default?

2. The whole setup still has a fairly high latency. Even over a direct 1Gbps
Ethernet link, there's still somewhere between 1 and 2 seconds of delay. I've
already reduced the queue duration on the sender, which helped a bit, but when I
reduce the receiver queues, I just get lots of corrupted frames. What would be
the correct way to reduce latency here?

Thanks for reading until here, hope you might have some answers for me :-)

Best regards, Florian
-- 
SENT FROM MY DEC VT50 TERMINAL

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20200224/f8d0db96/attachment.sig>