[gst-devel] rtp - mux or not to mux?

[Kai Vehmanen]

Hello all,

continuing the rtp-invasion of gstreamer-devel. :) Earlier I posted an 
high-level proposal about my gstsdpbin idea:

http://article.gmane.org/gmane.comp.video.gstreamer.devel/13940

Now I'm working on the details of the demuxing/switching part, in other 
words, how to support multiple codecs inside one stream of RTP packets 
(RTP session). This should be useful for various uses of RTP in 
gstreamer apps...

The problem is somewhat similar to mpeg/ogg demuxing, but with few key 
differences:

- an RTP stream will only contain one logical stream;
   the codec might change, but the packets will have different
   timestamps and sequence numbers (different codecs do
   not overlap timestamp wise, as happens with ogg/mpeg and
   other container streamer)
     => the reception pipeline is really 1:1 
- in most cases, you will know the possible subcomponents
   beforehand; and actually, with many codecs it's actually
   impossible to identify the codec type by looking at the RTP packets
     => the application has to specify which codecs, and
        pt<->codec mappings to use in parsing the incoming stream

I've been studying the current demuxers (and other matching elements in 
gstreamer), and it seems that this could be implemented in gstreamer 
either as:

1) rtpdemux -> 1:N demux element (like oggdemux for example)
     - connected to rtpbin/rtprecv's src pad
     - as auto-detection is not possible, client will have to
       setup pads beforehand (use PT=x for G711, PT=x for speex,
       and so on; static PTs are an exception here)
     - the mux will look at incoming packets, and push the
       them forward to the matching src pad
     - the pipeline:
       udpsrc -> rtprecv -> rtpdemux -> depayloader(s) -> decoder(s) -> sink
     - possible problems:
         - in the audio case, I guess "adder" wouldn't be enough to mux the
           multiple decoder paths together as it requires input from
           all sources before mixing
         - a modified version of "adder"? .. and same for video

2) rtp-switcher -> 1:1 element that can decode and depacketize multiple codecs
    - as (1), connected to rtpbin/rtprecv's src pad
    - creates and manages multiple depayloader+decoder elements
    - each received packet is pushed through exactly one
      depayloader+decoder combo
    - sink pad takes application/x-rtp, and pushes out
      encoded media (e.g. audio/x-raw-int) on its src pad
    - possible problems:
         - how to implement in gstreamer? any good example elements
           that are doing something like this?

Now any comments on which approach is best from gstreamer architecture 
point of view?

One problem common to both approaches is the jitter buffer code, now 
in gst-plugins-base-cvs/gst-libs/gst/rtp/gstbasertpdepayload.c. Ideally 
different codecs should share the same jitter buffer (so that change of
codec mid-stream wouldn't result in throwing out the whole jitter buffer), 
but this is pretty hard to realize with the current elements... or?

--