appsrc frame rate and glass2glass latency

[adam255]

Dear developers,

We use the appsrc with udp sink to deliver live video from camera to client
on TX2 with the following signal processing pipeline.

------------------------------------- begin signal processing pipeline
-------------------------------------
Server side:
SceneTime(t0) ->camera → (t1) Capture(at 5Hz) → BayerImgQueue →
SignalProcessing–> (t2) outputYuvImgQueue →
(t3)GstreamerPipeLineSendVideoToClient

GStreamer pipeline:
pipelineStr = “appsrc format=3 is-live=true !
video/x-raw,format=(string)I420, width=(int)” + std::to_string(w) +
“,height=(int)” + std::to_string(h) +
" , framerate=" + std::to_string(frameRate)+ “/ 1” +
" ! identity check-imperfect-timestamp=true" +
" ! nvvidconv" +
" ! omxh264enc control-rate=2 target-bitrate=4000000 tune=zerolatency" +
" ! video/x-h264,stream-format=(string)byte-stream ! h264parse ! rtph264pay"
+
" ! udpsink host=" + clientIp +" port=50000";

Client side:
UdpSrc → display (t4)
------------------------------- end signal processing pipeline
-------------------------------------------

Let’s define the corresponding timestamps and time intervals as follows.
t0: time stamp on a computer screen which is captured by camera
t1: time at which a frame y is just captured
t2: time at which the frame y is processed and written into the
ouputYuvImgQueue
t3: time at which the frame y is read from ouputYuvImgQueue by
need_data_callback()
t4:time stamp on receiver side screen
dt1 = t2-t1
dt2 = t3-t2
dt3 = t4-t0 (glass-to-glass latency)

Our experiments show that the glass-to-glass latency strongly depends on the
frameRate parameter in video/x-raw cap. For example
frameRate   dt1(ms)   dt2 range(ms)	 max dt3(ms)
5                  25           [700,850]             ~1290
6                  25           [270,300]              ~740 		
7                  25           [200,260]              ~690			
10                25           [150,200]              ~630    
15                25           [110,120]              ~550
20                25           [92,100]               ~530
30                25           [75,79]                 ~500

Note: when there is no new image in outputYuvImgQueue and
need_data_callback() is called, we just repeat the previous frame.

My Question is why the frameRate parameter affects dt2 so much? To void
this, what kind of parameter shall we tune?

Thanks,



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/