RTP/RTCP Support and Extension Capabilities of GStreamer

[Mario Montagud Climent]

Dear Gstreamer developers,

My name is Mario Montagud. I am a PhD student at Politechnic  
University of Valencia (UPV) in Spain.

My PhD work is about Multimedia Communications and Synchronization  
Techniques. Concretely, our research group is designing an  
Inter-Destination Multimedia Synchronization (IDMS) proposal that aims  
to synchronize the playout of media streams  across separated clients  
(which can be placed in different networks, with different network  
latency). For that purpose, we take advantage of the feedback and  
extension capabilities of RTP/RTCP protocols.

In the initial stages of my PhD, I have implemented this IDMS approach  
in a network simulator (NS-2), and I?ve added some enhancements to  
that solution (reference [Bor11]).

Our goal is to synchronize a RTP network stream across multiple  
geographically separated clients. On the one hand, a new RTCP report  
block (RTCP XR for IDMS block) has been specified to inform the source  
(or a sync manager) about receipt and playout times for specific RTP  
media packets (for that purpose all the participants must share a  
common available clock source, e.g. NTP, GPS, or other solutions ?) at  
the receiver side (in a compound RTCP packet). On the other hand a new  
RTCP packet type (RTCP IDMS packet) has been defined to send playout  
setting instructions to the sync clients (if an out of sync situation  
is detected by the sync manager). See [Bra11].

My next objective is to implement and test that IDMS proposal in a  
real media framework/player. So, I?m searching for the most  
appropriated solution for it.
I am a newbie in GStreamer. For that reason, I send this mail asking  
for your recommendation. During last days, I?ve reading the Gstreamer  
documentation (GStreamer Plugin Writer?s Guide, and GStreamer  
Application Development Manual) to get a better understanding of this  
framework.

I?ve seen Gstreamer supports RTP/RTCP functionalities  
(/gst-plugins-good-plugins-gstrtpbin.html) and also lip-sync mechanisms.

I?ve also found a post from Thomas Roos to the Gstreamer Developers  
list about the synchronisation of audio playback (multi room feature)  
on multiple receivers. He did not succeed on the configuration of such  
multi room sync scenario. However, he told me that Gstreamer is the  
right way to do it! ;)
Thus, I would like to ask four your recommendation/suggestions before  
I begin with the implementation efforts.

First, In order to implement our IDMS solution, I need to implement  
the new RTCP packet types and block reports defined in [Bra11].

In our IDMS solution, we tackle the sync problem by dividing it in two  
main phases:

1)	To ensure that all the receivers initiate the playout of the media  
stream at the same time.

For that purpose, I need to implement some mechanism to ensure that  
all the clients will buffer the incoming media units (encapsulated in  
RTP packets) until a global initial playout instant (this info will be  
received in a new RTCP packet type for IDMS from the server) is  
reached. Then, the playout process must start.

2)	To maintain the playout processes between distributed clients in a  
synchronized way during the streaming session lifetime.
During the session, receivers will send a new RTCP XR blocks including  
arrival/presentation times for a specific RTP packet (e.g. the last  
timestamp received from one RTP packet containing a media unit ? video  
frame-). A sync manager (e.g. the server) will gather this reports  
from all the clients and, if an out of sync is detected (exceeding an  
asynchrony threshold), it will send a new RTCP packet type for IDMS  
including playout setting instructions to the clients.

Once the clients receive this packet, they will have to trigger  
playout adjustment to get in sync according to the playout info  
included in that control message.

I have seen the ?ts-offset? calculation for lip-sync purposes in  
?gstrtpbin? module. By the way, how are the required playout  
adjustments managed in the lip sync mechanism? (smooth) playback speed  
variation? skips/pauses?

I think the most difficult task will be the management of the playout  
processes in each media client to get in sync (e.g. to report on  
presentation time for a specific RTP packet ?e.g. the last one  
encapsulating a video frame-, and to manage the necessary playback  
adjustments. Can I access to the playback buffer to know about  
presentation times of a specific video frame and the current playback  
buffer occupancy?

I apologize for the extension of the mail, but I will appreciate very  
much to get some initial guidelines from Gstreamer experts before  
starting with the design and implementation efforts.

Thank you very much in advance.

Regards from Spain,

Mario

REFERENCES: (If you are interested, and don?t have access to them,  
please, tell me)

[Bor08] F. Boronat, J.C. Guerri, and J. Lloret, ?An RTP/RTCP based  
approach for multimedia group and inter-stream synchronization?,  
Multimedia Tools and Applications Journal, Vol. 40 (2), 285-319, June  
2008.

[Bor11] F. Boronat, M. Montagud, and V. Vidal, ?Smooth Control of  
Adaptive Media Playout to Acquire IDMS in Cluster-based Applications?,  
IEEE LCN 2011, Bonn, October 2011.

[Bra11] R. V. Brandenburg, H. Stokking, M.O. V. Deventer, Boronat, F.  
M. Montagud, K. Gross, ?RTCP for Inter-Destination Media  
Synchronization?, IETF Internet Draft, October 2011.