AV sync with hardware decoder and segment events

[Sachs Jürgen]

Hallo together,
We have some problems to get AV in sync with our hardware decoders with some kind MP4 files.
 
We have implemented audio- and video sinks based on a hardware decoder. This decoder(s) have a (common) clock (STC) . When I feed audio and video buffers with timestamps (PTS), the hardware will output the decoded frames according to this STC. 
To build the PTS for the hardware from the buffer PTS we
-          In SEGMENT event store the segment in the element and reset the hardware STC to 0 (both for audio and video)
-          In push() convert the PTS for the decoder with 
o   PTS = gst_segment_to_running_time (&thiz->seg, GST_FORMAT_TIME, GST_BUFFER_PTS (buffer));
... and send the buffer with this new PTS to hardware
 
Is this correct?
 
As written above, this works for most streams, but we found some, where AV is not in sync. (compared to the playback of this streams with VLC or Windows Mediaplayer). The special of this streams seems to be the elst-box in the MP4 file. 
Mostly I get two segment events before the first video buffer, for example
time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:00.000000000, duration 99:99:99.999999999
time segment start=0:00:00.083333333, offset=0:00:00.000000000, stop= 0:05:00.083333333, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:00.083333333, duration 99:99:99.999999999
render l=        1120 t=0:00:00.083333333 flags=00004040 free=         100
In the wrong case, I get 3 events like this
time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop=99:99:99.999999999, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:00.000000000, duration 99:99:99.999999999
time segment start=0:00:00.000000000, offset=0:00:00.000000000, stop= 0:00:00.080000000, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.000000000, base=0:00:00.000000000, position 0:00:00.000000000, duration 99:99:99.999999999
time segment start=0:00:00.080000000, offset=0:00:00.000000000, stop= 0:04:30.400000000, rate=1.000000, applied_rate=1.000000, flags=0x00, time=0:00:00.080000000, base=0:00:00.000000000, position 0:00:00.080000000, duration 99:99:99.999999999
render l=       34461 t=0:00:00.080000000 flags=00000040 free=         100
 
Thanks for any help or confirmation, that I´m not totally wrong. I´m not sure, if my STC/PTS calculation is correct or if possibly the is an issue in qtdemux.
 
Kind regards
/Juergen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20170714/9c83e10b/attachment.html>