[gst-devel] MPEG over RTP

[Ralph Meijer]

On Wed, Jul 16, 2003 at 09:15:18AM -0700, Ted Gould wrote:
> On Wed, 2003-07-16 at 08:07, Ralph Meijer wrote:
> > A way to fix this problem would be to use the transport order of the pictures
> > for generating the buffer timestamps, but then I have to communicate the
> > timestamp in some other way, for example by extending the payload of the
> > buffers with a new field containing the timestamp. But then I might as well
> > create the whole RTP packet, right?
> > 
> > So my idea would be to combine the two components into one that takes
> > an MPEG stream as it input and outputs RTP packets.
> 
> I don't understand why you could recreate the timestamps on the receive
> side rather easily.  You'd have to cache the data for a couple of anchor
> frames, but I'm guessing that since your going over an IP network a
> little delay won't hurt you.  I'll go with the example that you gave 
> (I'm going to divide all your numbers by 40)

Ah, maybe I didn't make clear what my problem is. Recreating timestamps
on the receiving side is not an issue. First of all, the mpeg decoder
doesn't need the timestamps that are put in the RTP packet, and second
of all, well, they are already in the RTP packet ;-).

A bit of theory:

An MPEG stream doesn't carry timestamps itself for each picture. An MPEG
stream can be viewed as a stack:

.-------------------.
|  Sequence header  |
+-------------------+
|     GOP header    |
+-------------------+
|   Picture header  |
+-------------------+
|    Slice header   |
+-------------------+
|     Macroblocks   |
`-------------------'

So, a sequence has a string of GOPs (Group of Pictures), a GOP is composed of
an arbitrary and possibly irregular number of pictures, a picture is composed
of (horizontal) slices and the macroblocks contain the actual picture data.
One thing I'd like to note here is that GOP sizes can be irregular, for
example because scene changes.

The sequence header contains information like the framerate. Each GOP header
has a timestamp (hours, minutes, seconds, frame# in that second). So
the timestamp of the first picture in that GOP in display order is
3600*hours + 60*minutes + seconds + frame#/framerate.

RFC 2250 requires the RTP timestamp to be filled with the presentation
timestamp, so this has to be created in the way I described above. So
each RTP packet (which can contain a fragment of the data belonging
to a certain picture) carries this timestamp. But they are not monotonically
increasing.

Now my problem, hopefully a little clearer:

As I said, my first try was to communicate the picture timestamp, to be
put in the RTP headers, as buffer timestamp. And also the RTP buffers
going to the udpsink would have this timestamp.

What I have noticed is that this happens: GStreamer sees a buffer flowing
to the udpsink with timestamp 80 (in milliseconds). The udpsink's chain
function is called with the buffer and all goes well. Now the following
buffer is pushed by the rtpmpegenc. This also has timestamp 80, which is
quickly given to udpsink. Now a buffer comes along with timestamp 0. "Hrm,
WTF?" Gstreamer thinks, 'this one is too late, let's quickly process that
one too!" This all goes on until the buffer with timestamp 200 is
pushed, and GStreamer happily waits for 120 milliseconds before handing
it to udpsink.

So looking at the udp traffic comming out of udpsink, there are bursts
of data comprising 3 pictures. This means that the receiving end
has to quickly process the incoming burst of data, otherwise the
receive buffer overflows (even when using the loopback network). Especially
slower machines suffer from this, but it could also happen that devices
along the route start dropping data, since UDP traffic is best effort,
no retransmissions.

To make the udp traffic less bursty, GStreamer should be pursuaded to
deliver the RTP packets more evenly. To accomplish this, I thought
I might use fake timestamps for the buffers, based on the GOP timestamp
and the count of the picture in *tranport* order:

Pictures:         I    B    B    P    B    B    P    B    B
 
 Transport order:  0    1    2    3    4    5    6    7    8
 Display order:    2    0    1    5    3    4    8    6    7
 Timestamp:       80    0   40  200  120  160  320  240  280
 Fake Timestamp:   0    40  80  120  160  200  240  280  320

As I said in my other mail, I now have to communicate 2 timestamps
between the rfc2250enc and rtpmpegenc component: the timestamp in
display order and the fake one in transport order. Combined
with the need to fill the other MPEG-specific RTP headers, I'm /almost/
making the whole RTP packet in the rfc2250enc component. So that's
why I suggested to merge the two.

-- 
Groetjes,

Ralphm