Working of UDP source timeout feature/ RTP/RTCP/ timestamping video frames

[vk_gst]

Hi Nicolas and others,

Please bear the long post. 

With regards to your suggestion to look into RTCP for the use case,  I found
out that RTCP sends statistics packets only as a part of the total data
transfer. May be this is intended for detecting the congestion over the
network over a period of time. 

Now this approach may sound goofy or not practical, but I was wondering : 
1. if I can get the timestamps of the buffer sent from Tx, and 
2. use these timestamps in addition to the sequence number of buffers(RTP)
at the Rx  
and predict the network congestion/delay in receiving the next buffers.

This would mean  timestamps for each video frame (or timestamps for each RTP
packets). 

Why I want to do this : 
1. I want to insert an IMU frame(custom visualization) in between the live
video frames, whenever the network link breaks, congestion detected, or a
delay is predicted. 
2. The worst case would be to insert an IMU frame between alternate video
frames. eg for a 30 fps video 
 and an interval of 33ms the following: 

                V|I|V|I|V|I|V|I|V|I|V|I|V|I|

where V -video frame and I - IMU frame. 

However, to be practical I am hoping to insert an IMU frame for every 100ms,
in case the link breakdown is detected on such a precision.

Now I have no idea, if this makes sense to take the timestamps of each
frame/buffer and then at the Rx  calculate the delay and predict the
congestion and then insert an IMU frame. Has this been done by anyone in any
application so far? How is the synchronisation between IMU and video
achieved in mobile phones? 

1. Can anyone point me to some direction or current applications that
utilize some of these features?
2. I tried accessing the PTS and DTS of the frames, but these timestamps do
not give me the time at which the packets were sent from Tx. So I cannot
calculate the delay (Timestamp Rx - Timestamp Tx) in receiving them. Which
timestamps should I be looking for at the Rx side,  to measure the delay in
timestamps, since I need a common reference to the clock at both Rx and Tx? 
3. The RTCP does provide an absolute clock, but then the RR/SR packets for
RTCP are only sent a fraction of the total packets sent. So it is not for
individual frames as well. 
4. Please feel free to add any other suggestions/approach when it comes to
handling such an application



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/