Playback of my RTSP streamed video choppy

Tue Apr 14 08:47:35 PDT 2015

After grinding through the rtph264pay and basesink source code and evaluating runtime behavior, I found one source of my problem to be non-intuitive.

In my application I parse the h264 video data received from the camera into NAL units, and one I have a NAL unit, I gave it a time stamp relative to the camera's frame rate.
At every new IDR, the camera inserts a SPS/PPS NAL unit.  I was submitting a timestamped buffer to the appsrc that included the SPS/PPS and IDR NAL units.  

This caused the RTP packetizer to skip one RTP frame slot every IDR.   So at 10fps, the expected increment in RTP time is 9000 for every frame, but when my buffer with the SPS/PPS and IDR was processed, the packetizer would tag the IDR with a RTP time difference of 18000.

To resolve this, I simply put the SPS, PPS, and IDR into their own buffer that is pushed onto the appsrc.  I set the timestamp on the SPS and PPS to be the same as the last PFrame emitted, and the IDR with a 100ms increment (at 10FPS), and the packetizer is now successfully organizing the RTP.

All is not well however, the timing of the packets getting onto the network (as observed by Wireshark) has jitter, where there will be delays that range from 100ms (expected) to 1.2sec (bad).  So I am looking at what can be done to optimize getting these packets emitted with more consistent timing.  VLC is complaining constantly about packets being late, and my stream is still choppy.  Better, but still choppy.

-----Original Message-----
From: gstreamer-devel [mailto:gstreamer-devel-bounces at lists.freedesktop.org] On Behalf Of Tim Müller
Sent: Thursday, April 09, 2015 3:44 AM
To: Discussion of the development of and with GStreamer
Subject: Re: Playback of my RTSP streamed video choppy

On Wed, 2015-04-08 at 22:30 +0000, Kerry Calvert wrote:

Hi Kerry,

> My pipeline is as follows: appsrc is-live=true name=h264appsrc
> blocksize=8192 ! h264parse ! rtph264pay name=pay0 pt=96
> config-interval=5 perfect-rtptime=false
> 
> My code putting data into the app src is scanning the input video 
> stream and breaking it into NAL units.  Each NAL unit is then put into 
> a GSTBUFFER, and the PTS timestamp assigned to the GSTBUFFER is the 
> timestamp of the previous GSTBUFFER + incremental time per frame rate.
> So for 5FPS, the timestamps would go 0, 200ms, 400ms, 600ms,800ms, 
> etc.  This algorithm gives us the ability to clearly organize the 
> video data into frames to be played back at a given framerate.

So you are setting the timestamps on the buffers you feed into the pipeline yourself. Thing is, this is a live pipeline, so this might be a bit problematic.

Two things to try:

a) don't set timestamps yourself and use

  appsrc is-live=true do-timestamp=true min-latency=500000000

just to see if it makes any difference.

b) add a queue after appsrc (then you can feed data ahead of time; you may still want to set some min-latency though, to avoid things being dropped at the beginning.)

In a live pipeline the data captured at the source should be timestamped with the current running time of the pipeline clock, and there would typically also be some >0 latency. I suspect what's happening in your case is that you "under-timestamp" your data, so the sink ends up discarding a lot of buffers because it considers them too late. You can figure out if that's what's happening by looking at the
GST_DEBUG=basesink:6 log of the sending (multi)udpsink.

http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/design/part-latency.txt

> I feel like something is inefficient in GStreamer’s rtp packetizing or 
> in its ability to push packets onto the network at the desired timing.
> I have the same problem whether I run my camera at 10fps or 5fps.
> 

If it also happens at 5fps it's probably not a matter of efficiency, but something systematically wrong, like the timestamping + latency configuration of the pipeline.

> I am looking for some hints for how to determine where this bottleneck 
> is.  I need to determine if the problem is indeed in GStreamer or is 
> in the network layer.

Looking at the basesink logs should tell you if it drops buffers before sending them. A wireshark capture of the data sent (on the sender side and the receiver side) should also tell you if all data is sent/received or not (by looking at the rtp sequence number and timestamps etc.).

Cheers
 -Tim

--
Tim Müller, Centricular Ltd - http://www.centricular.com

_______________________________________________
gstreamer-devel mailing list
gstreamer-devel at lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel