packetized vs bytestream in rtp

[Chuck Crisler]

Yes, I have h264parse before rtph264pay. Yes, rtph264pay can/will merge FUs.

The overall process is this: iPad/H264/RTP -> gstreamer RTP to multicast
MP2T ->gstreamer mutlticast MP2T to RTP -> iPad/RTP

Unfortunately, the MP2T portion is absolutely required due to legacy
systems. :-(((

I can't see the internal buffer data, but h264parse is finding a start code
around 900 bytes into the incoming buffers. The original RTP stream had
about 2200 bytes of payload data (over about 3 packets), so I know that
this start code is artificial. The next buffer into h264parse has a
first_mb_in_slice of around 200+. However, at that point h264parse has
already pushed the previous buffer downstream (to rtph264pay). rtph264pay
determines that the buffer it receives is complete (how would it know it
isn't?) so it processes it as a complete packet that fits within the MTU
and signals it as such. The only clue that there is more data for the frame
is that the first_mb_in_slice on later buffers is greater than 0. I think
that I need h264parse to accumulate multiple input buffers before pushing
anything to rtph264pay until the slice number is 0. When it does, the
buffer should be around 2k bytes in my current testing. Here is a brief log
snippet to illustrate the problem. I have confirmed this with packet
captures.

h264parse gsth264parse.c:2569:gst_h264_parse_chain:<H264Parse>
received buffer of size 720
h264parse gsth264parse.c:1964:gst_h264_parse_chain_forward: Next NAL
Unit Pos: -1  <== I added this msg, it shows that there isn't a start code
yet
gsth264parse.c:2014:gst_h264_parse_chain_forward:<H264Parse> NAL type:
5, ref_idc: 3
gsth264parse.c:2033:gst_h264_parse_chain_forward:<H264Parse> first MB
in slice: 0, slice type: 7 <== first part of IDR
gsth264parse.c:2055:gst_h264_parse_chain_forward:<H264Parse> we have
an I slice
h264parse gsth264parse.c:2569:gst_h264_parse_chain:<H264Parse>
received buffer of size 170 <== received another buffer segment
h264parse gsth264parse.c:1964:gst_h264_parse_chain_forward: Next NAL
Unit Pos: 890 <== second segment had a start code - don't know where from
but guess MP2T demux, still not the end of the H264 frame though
h264parse gsth264parse.c:2014:gst_h264_parse_chain_forward:<H264Parse>
NAL type: 5, ref_idc: 3
h264parse gsth264parse.c:2033:gst_h264_parse_chain_forward:<H264Parse>
first MB in slice: 0, slice type: 7 <== starts back at the beginning, I
will have to deal with this
h264parse gsth264parse.c:2055:gst_h264_parse_chain_forward:<H264Parse>
we have an I slice
h264parse gsth264parse.c:2223:gst_h264_parse_chain_forward:<H264Parse>
pushing buffer 0xf6f02de8, size 890, ts 0:00:00.066733333 <== NO! We really
aren't done. Trust me!
h264parse gsth264parse.c:1964:gst_h264_parse_chain_forward: Next NAL
Unit Pos: -1
h264parse gsth264parse.c:2014:gst_h264_parse_chain_forward:<H264Parse>
NAL type: 5, ref_idc: 3
h264parse gsth264parse.c:2033:gst_h264_parse_chain_forward:<H264Parse>
first MB in slice: *228*, slice type: 7 <== I told you that we weren't
finished!

At this time rtph264pay has already sent the original buffer out as a
completed RTP packet with the marker bit set. This causes the decoder
(using iDoubs on an iPad) to fail. Gstreamer can handle this condition
properly. However, this is effectively an error condition, even though the
spec uses 'should' concerning the marker bit rather than 'shall'. Everyone
relies on the marker bit. But that doesn't matter. The required ending
(display on an iPad) doesn't work (yet).

Chuck

On Thu, Mar 28, 2013 at 3:08 PM, Olivier Crête
<olivier.crete at collabora.com>wrote:

> On Thu, 2013-03-28 at 14:54 -0400, Chuck Crisler wrote:
> > Actually, as I have been working on this over time I have come to
> > realize that the current gstreamer (or any RTP code) cannot possibly
> > work. There is important information in the first RTP stream that is
> > lost in the conversion to MP2T. Specifically the fact that there are
> > multiple RTP packets required for the frame which is signaled via the
> > FU-A construct. However, that seems to be OK for the MP2T. When the
> > conversion back to RTP is made, it is only processing those MP2T
> > packets that were created from each single RTP packet in the original
> > stream. I don't think that there is any way in the normal scheme to
> > correct that. I think that to correct it I will need to modify
> > h264parse (my first element in the conversion back to RTP) to collect
> > input buffers until the first_mb_in_slice == zero. Then the previous
> > data can be processed and pushed out as a single buffer for rtph264pay
> > to process. I have seen other places in the code of both elements that
> > make that same assumption, so it seems safe to me.
>
> I'm not sure I understand exactly what you mean, but rtph264depay will
> merge back FU-A and FU-B packets into a single GStreamer buffer (or drop
> them completely if any part is missing), if you control the encoding,
> you are strongly advised to make the NAL unit smaller than the MTU.
>
> I think you want to collect all of the buffers in an access unit (where
> first_mb_in_slice == zero), then rtph264depay can do that for you if the
> caps afterwards having "alignment=au". You should never need to have
> h264parse after rtph264depay.
>
> --
> Olivier Crête
> olivier.crete at collabora.com
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20130328/25a1d2a1/attachment.html>