how to understand sources of latency in my RTSP client pipeline?

[Dan Williams]

Hi All,

I'm trying to minimize the latency in this pipeline:

  gst-launch-1.0 rtspsrc latency=<value> location=<URI> \
  ! rtph264depay ! vaapiparse_h264 ! vaapidecode \
  ! vaapipostproc width=1824 deinterlace-mode=disabled \
  ! vaapisink fullscreen=true

This runs on an Atom E3815 and has plenty of CPU and graphics power
for this task. OS is either Ubuntu 14.04 or a Yocto build.

I'm using the software as of:
  commit 8d835ec400819140e5fc8532a90010c694e1212b
  Date:   Sun Nov 23 05:45:24 2014 -0300

The RTSP source is an IP camera sending out H.264 Main Level streams
via UDP transport. Both sender and receiver are plugged into the same
1Gbps switch. The camera can produce two different streams, one at
1920x1080 at about 150KB/s and another at 432x240 at about 30KB/s.

I'm measuring latency by pointing the IP camera at a running
high-precision clock and then taking photos which include both the clock
and the graphics display at the same time.

Using the high resolution stream, if I set the latency <value> to the
default of 2000ms, I measure an actual latency of about 2400ms. I am
able to reduce the set latency <value> to 500ms and get a measured
latency of about 900ms. If I reduce the set latency <value> to 400ms
or below, it drops frames and prints this:

  WARNING: from element /GstPipeline:pipeline0/GstVaapiSink:vaapisink0: A lot of buffers are being dropped.
  Additional debug info:
  gstbasesink.c(2794): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstVaapiSink:vaapisink0:
  There may be a timestamping problem, or this computer is too slow.

Using the low resolution stream, if I set the latency <value> to the
default of 2000ms, I measure an actual latency of about 2600ms. But,
if I reduce the set latency <value> to anything below about 1500ms, I
get the same printed error messages and instead of merely dropping
frames I get a very strange graphics display where it freezes every 2
seconds (the I frame period) and then quickly displays some
intermediate frames before freezing again.

My immediate goal is to try to get the low resolution stream latency
to be at least as good as the high resolution stream latency.

At first I thought that some difference between the two streams would
be apparent, but this has not been the case. I've sniffed the network
traffic, looked at the streams with "ffprobe -show_frames" and don't
see any differences that could explain this. They are encoded the same
way (no B frames, I frames every 2 seconds) and have the same
timestamps.

I've experimented with different settings for many rtspsrc and
vaapisink parameters without success.

I've been looking at the logs generated via "GST_DEBUG=*:6" with
particular attention to GST_PERFORMANCE and QOS messages, but I am no
closer to figuring this out.

I'm looking for some advice on what to log and more importantly, how
to interpret those logs to understand what is going on.

Thanks!
Dan