<div dir="ltr">Decoding timestamps are currently inconsistent through GStreamer, making it impossible in some situations to mux proper streams. <div><br></div><div>1. GStreamer doesn't support negative DTS or DTS outside current segment. Why is this necessary? Say you have sequence such as <span style="color:rgb(51,51,51);white-space:pre-wrap">IPBPP </span><span style="color:rgb(51,51,51);white-space:pre-wrap">(PTS=0,2,1,3,4 DTS=-1,0,1,2,3). B-frame in the middle has PTS of 1, but needs two frames decoded before it can be displayed (thus they need to have DTS = -1, 0).</span></div>
<div><span style="color:rgb(51,51,51);white-space:pre-wrap"><br></span></div><div style><span style="color:rgb(51,51,51);white-space:pre-wrap">When muxing, timestamps are converted to running time. When PTS running starts from 0, in some cases DTS must be before that (example above). Right now gst_segment_to_running_time just throws the value away. That means we end up with bunch of samples with missing DTS at the beginning of file.</span></div>
<div style><span style="color:rgb(51,51,51);white-space:pre-wrap"><br></span></div><div style><span style="color:rgb(51,51,51);white-space:pre-wrap">2. Inconsistent handling. Some parts of GStreamer assume that DTS = PTS for keyframes. Now in order to do that, the DTS needs to be shifted, resulting in DTS > PTS for some frames. This doesn't seem right to me, as you can't display frame first and then decode it.</span></div>
<div style><span style="color:rgb(51,51,51);white-space:pre-wrap"><br></span></div><div style><span style="color:rgb(51,51,51);white-space:pre-wrap">For example, x264 enc is currently doing this. It offsets DTS so that it is never negative (and matches PTS on keyframes), but this means it produces DTS > PTS. What is worse, it is throwing away information. The exact DTS values are important in some cases, i.e. for hardware decoders and MPEG TS. If the DTS aren't right the decoding buffer can underflow, etc. In current situation, even if the muxer would want to compensate for the shift (which it doesn't), it can't because it doesn't know how big the shift should be.</span></div>
<div style><span style="color:rgb(51,51,51);white-space:pre-wrap"><br></span></div><div style><font color="#333333"><span style="white-space:pre-wrap">Potential Solutions:</span></font></div><div style><font color="#333333"><span style="white-space:pre-wrap"><br>
</span></font></div><div style><font color="#333333"><span style="white-space:pre-wrap">1. Treat GstClockTime as signed. Change GST_CLOCK_TIME_NONE to something other than -1. Allow gst_segment_to_running/stream_time return negative value for DTS. Stop assuming that PTS=DTS for keyframes. </span></font></div>
<div style><font color="#333333"><span style="white-space:pre-wrap"><br></span></font></div><div style><font color="#333333"><span style="white-space:pre-wrap">This would be very nice IMO, but not very feasible I'm afraid, as changing the GST_CLOCK_TIME_NONE value right now is probably out of question.</span></font></div>
<div style><font color="#333333"><span style="white-space:pre-wrap"><br></span></font></div><div style><font color="#333333"><span style="white-space:pre-wrap">2. Add DTSOffset sticky serialized event. </span></font></div>
<div style><font color="#333333"><span style="white-space:pre-wrap"><br></span></font></div><div style><font color="#333333"><span style="white-space:pre-wrap">This would be an ugly kludge, but it would solve the issue of not being able to mux proper streams. It would however requre all elements that produce DTS (encoders, demuxer) and consume DTS (muxers, some decoders) to account for the offset.</span></font></div>
<div style><font color="#333333"><span style="white-space:pre-wrap"><br></span></font></div><div style><font color="#333333"><span style="white-space:pre-wrap">Every time an encoder/demuxer would produce DTS that it is before segment start, it would offset it (and all DTS that follow) so that it is within segment and send DTSOffset event so that downstream elements are able to restore original DTS.</span></font></div>
<div style><br></div><div style><font color="#333333"><span style="white-space:pre-wrap">This shouldn't need any API breaks as far as I can tell. It also shouldn't make anything worse for elements that don't handle the event (because the DTS are broken anyway), but it adds additional complexity.</span></font></div>
<div style><font color="#333333"><span style="white-space:pre-wrap"><br></span></font></div><div style><font color="#333333"><span style="white-space:pre-wrap">Or perhaps someone can think of a better solution?</span></font></div>
<div style><font color="#333333"><span style="white-space:pre-wrap"><br></span></font></div><div style><font color="#333333"><span style="white-space:pre-wrap">Cheers,</span></font></div><div style><font color="#333333"><span style="white-space:pre-wrap">Matej</span></font></div>
</div>