DTS being broken in GStreamer

[Matej Knopp]

Decoding timestamps are currently inconsistent through GStreamer, making it
impossible in some situations to mux proper streams.

1. GStreamer doesn't support negative DTS or DTS outside current segment.
Why is this necessary? Say you have sequence such as IPBPP (PTS=0,2,1,3,4
DTS=-1,0,1,2,3). B-frame in the middle has PTS of 1, but needs two frames
decoded before it can be displayed (thus they need to have DTS = -1, 0).

When muxing, timestamps are converted to running time. When PTS running
starts from 0, in some cases DTS must be before that (example above). Right
now gst_segment_to_running_time just throws the value away. That means we
end up with bunch of samples with missing DTS at the beginning of file.

2. Inconsistent handling. Some parts of GStreamer assume that DTS = PTS for
keyframes. Now in order to do that, the DTS needs to be shifted, resulting
in DTS > PTS for some frames. This doesn't seem right to me, as you can't
display frame first and then decode it.

For example, x264 enc is currently doing this. It offsets DTS so that it is
never negative (and matches PTS on keyframes), but this means it produces
DTS > PTS. What is worse, it is throwing away information. The exact DTS
values are important in some cases, i.e. for hardware decoders and MPEG TS.
If the DTS aren't right the decoding buffer can underflow, etc. In current
situation, even if the muxer would want to compensate for the shift (which
it doesn't), it can't because it doesn't know how big the shift should be.

Potential Solutions:

1. Treat GstClockTime as signed. Change GST_CLOCK_TIME_NONE to something
other than -1. Allow gst_segment_to_running/stream_time return negative
value for DTS. Stop assuming that PTS=DTS for keyframes.

This would be very nice IMO, but not very feasible I'm afraid, as changing
the GST_CLOCK_TIME_NONE value right now is probably out of question.

2. Add DTSOffset sticky serialized event.

This would be an ugly kludge, but it would solve the issue of not being
able to mux proper streams. It would however requre all elements that
produce DTS (encoders, demuxer) and consume DTS (muxers, some decoders) to
account for the offset.

Every time an encoder/demuxer would produce DTS that it is before segment
start, it would offset it (and all DTS that follow) so that it is within
segment and send DTSOffset event so that downstream elements are able to
restore original DTS.

This shouldn't need any API breaks as far as I can tell. It also shouldn't
make anything worse for elements that don't handle the event (because the
DTS are broken anyway), but it adds additional complexity.

Or perhaps someone can think of a better solution?

Cheers,
Matej
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20130424/d30f86bd/attachment.html>