Framerate calculation for qtdemux video source

Tue Jun 20 21:28:49 UTC 2017

Hello,

I’ve got a query relating to the implementation of frame rate calculation within qtdemux.c gst_qtdemux_configure_stream()

I’m using a fragmented mp4 file as input to the qtdemux element. Caps events are regularly being output from the element for the video stream being demultiplexed. The framerate in these caps events is being calculated as per this comment in qtdemux.c:

      /* Calculate a framerate, ignoring the first sample which is sometimes truncated */

The problem I am having is the app I’m writing depends on a fixed and accurate frame rate on the video source caps, but what I see is that if I feed in video at 25fps, the initial and subsequent caps I get out out fluctuate somewhere below 25fps (based on the calculation above).

Here is an example which works:

gst-launch-1.0 videotestsrc ! x264enc ! capsfilter caps=video/x-h264,framerate=\(fraction\)25/1 ! mp4mux fragment-duration=2000 ! qtdemux name=demux demux.video_0 ! capsfilter caps=video/x-h264 ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

And here it fails to negotiate as I force the output to be 25fps (which is exactly what I’m feeding in to the demuxer):

gst-launch-1.0 videotestsrc ! x264enc ! capsfilter caps=video/x-h264,framerate=\(fraction\)25/1 ! mp4mux fragment-duration=2000 ! qtdemux name=demux demux.video_0 ! capsfilter caps=video/x-h264,framerate=\(fraction\)25/1 ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Redistribute latency...
ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(2950): gst_base_src_loop (): /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...

I’ve included detailed logs below where I can see the issue happening e.g. "framerate=(fraction)49/2 not accepted"

I would have thought that the frame rate would be available in some form of media descriptor atom and therefore it wouldn’t need to be calculated?

Am I using Gstreamer incorrectly? Am I expecting something which isn’t possible? Or is it the case that parsing the correct atom in the quicktime fragments just needs to be implemented?

What about just updated the calculation to not ignore the first sample?

I’m using GStreamer 1.10.4

Thanks,
Nick

0:00:03.588554716     1      0x126ef70 DEBUG                qtdemux qtdemux.c:7570:gst_qtdemux_configure_stream:<demux> Calculating framerate, timescale 2500 gave fps_n 49 fps_d 2
0:00:03.588584225     1      0x126ef70 DEBUG        GST_PERFORMANCE gstcaps.c:178:_gst_caps_copy:video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)2, profile=(string)high, codec_data=(buffer)01640014ffe1001d67640014acd94141fb016a0c020b4a000003000200000300651e28532c01000568ebecb22c, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1 doing copy 0x7ff894002b70 -> 0x7ff894c012d0
0:00:03.588593725     1      0x126ef70 DEBUG        GST_PERFORMANCE gstminiobject.c:315:gst_mini_object_make_writable: copy GstCaps miniobject 0x7ff894002b70 -> 0x7ff894c012d0
0:00:03.588633077     1      0x126ef70 DEBUG                qtdemux qtdemux.c:7590:gst_qtdemux_configure_stream:<demux> video size 320x240, target display size 320x240
0:00:03.588644885     1      0x126ef70 DEBUG                qtdemux qtdemux.c:7593:gst_qtdemux_configure_stream:<demux> par 1:1
0:00:03.588652414     1      0x126ef70 DEBUG               GST_PADS gstpad.c:1849:gst_pad_set_event_function_full:<demux:video_0> eventfunc for set to 0x7ff89b6c1880
0:00:03.588659418     1      0x126ef70 DEBUG               GST_PADS gstpad.c:1929:gst_pad_set_query_function_full:<demux:video_0> queryfunc set to 0x7ff89b6c12f0
0:00:03.588665901     1      0x126ef70 DEBUG               GST_PADS gstpad.c:1069:gst_pad_set_active:<demux:video_0> pad was active in push mode
0:00:03.588673222     1      0x126ef70 DEBUG               GST_CAPS gstpad.c:2669:gst_pad_get_current_caps:<demux:video_0> get current pad caps video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)2, profile=(string)high, codec_data=(buffer)01640014ffe1001d67640014acd94141fb016a0c020b4a000003000200000300651e28532c01000568ebecb22c, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1
0:00:03.588698320     1      0x126ef70 DEBUG                qtdemux qtdemux.c:7710:gst_qtdemux_configure_stream:<demux> setting caps video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)2, profile=(string)high, codec_data=(buffer)01640014ffe1001d67640014acd94141fb016a0c020b4a000003000200000300651e28532c01000568ebecb22c, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)49/2
0:00:03.588718327     1      0x126ef70 INFO               GST_EVENT gstevent.c:808:gst_event_new_caps: creating caps event video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)2, profile=(string)high, codec_data=(buffer)01640014ffe1001d67640014acd94141fb016a0c020b4a000003000200000300651e28532c01000568ebecb22c, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)49/2
0:00:03.588739288     1      0x126ef70 DEBUG              GST_EVENT gstevent.c:305:gst_event_new_custom: creating new event 0x7ff894bcf110 caps 12814
0:00:03.588793540     1      0x126ef70 DEBUG               GST_PADS gstpad.c:5101:store_sticky_event:<demux:video_0> notify caps
0:00:03.588807146     1      0x126ef70 DEBUG               GST_PADS gstpad.c:3860:check_sticky:<demux:video_0> pushing all sticky events
0:00:03.588812214     1      0x126ef70 DEBUG               GST_PADS gstpad.c:3792:push_sticky:<demux:video_0> event stream-start was already received
0:00:03.588819453     1      0x126ef70 DEBUG              GST_EVENT gstpad.c:5546:gst_pad_send_event_unchecked:<capsfilter1:sink> have event type caps event: 0x7ff894bcf110, time 99:99:99.999999999, seq-num 62, GstEventCaps, caps=(GstCaps)"video/x-h264\,\ stream-format\=\(string\)avc\,\ alignment\=\(string\)au\,\ level\=\(string\)2\,\ profile\=\(string\)high\,\ codec_data\=\(buffer\)01640014ffe1001d67640014acd94141fb016a0c020b4a000003000200000300651e28532c01000568ebecb22c\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ framerate\=\(fraction\)49/2";
0:00:03.588848541     1      0x126ef70 DEBUG               GST_CAPS gstutils.c:2983:gst_pad_query_accept_caps:<capsfilter1:sink> accept caps of video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)2, profile=(string)high, codec_data=(buffer)01640014ffe1001d67640014acd94141fb016a0c020b4a000003000200000300651e28532c01000568ebecb22c, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)49/2
0:00:03.589047936     1      0x126ef70 DEBUG                  query gstquery.c:674:gst_query_new_custom: creating new query 0x7ff894c01320 accept-caps
0:00:03.589061156     1      0x126ef70 DEBUG               GST_PADS gstpad.c:3929:gst_pad_query:<capsfilter1:sink> doing query 0x7ff894c01320 (accept-caps)
0:00:03.589075826     1      0x126ef70 DEBUG             capsfilter gstcapsfilter.c:341:gst_capsfilter_accept_caps:<capsfilter1> can intersect: 0
0:00:03.589084013     1      0x126ef70 DEBUG               GST_PADS gstpad.c:3952:gst_pad_query:<capsfilter1:sink> sent query 0x7ff894c01320 (accept-caps), result 1
0:00:03.589091821     1      0x126ef70 DEBUG                default gstutils.c:2988:gst_pad_query_accept_caps:<capsfilter1:sink> query returned 0
0:00:03.589098315     1      0x126ef70 DEBUG               GST_CAPS gstpad.c:5475:pre_eventfunc_check:<capsfilter1:sink> caps video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)2, profile=(string)high, codec_data=(buffer)01640014ffe1001d67640014acd94141fb016a0c020b4a000003000200000300651e28532c01000568ebecb22c, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)49/2 not accepted
0:00:03.589113830     1      0x126ef70 DEBUG               GST_PADS gstpad.c:5718:gst_pad_send_event_unchecked:<capsfilter1:sink> pre event check failed
0:00:03.589118318     1      0x126ef70 DEBUG               GST_PADS gstpad.c:3839:push_sticky:<demux:video_0> result not-negotiated, mark pending events
0:00:03.589123503     1      0x126ef70 DEBUG                qtdemux qtdemux.c:6698:gst_qtdemux_process_adapter:<demux> stream : avc1
0:00:03.589130054     1      0x126ef70 DEBUG               GST_PADS gstpad.c:3860:check_sticky:<demux:video_0> pushing all sticky events
0:00:03.589133598     1      0x126ef70 DEBUG               GST_PADS gstpad.c:3792:push_sticky:<demux:video_0> event stream-start was already received
0:00:03.589137616     1      0x126ef70 DEBUG              GST_EVENT gstpad.c:5546:gst_pad_send_event_unchecked:<capsfilter1:sink> have event type caps event: 0x7ff894bcf110, time 99:99:99.999999999, seq-num 62, GstEventCaps, caps=(GstCaps)"video/x-h264\,\ stream-format\=\(string\)avc\,\ alignment\=\(string\)au\,\ level\=\(string\)2\,\ profile\=\(string\)high\,\ codec_data\=\(buffer\)01640014ffe1001d67640014acd94141fb016a0c020b4a000003000200000300651e28532c01000568ebecb22c\,\ width\=\(int\)320\,\ height\=\(int\)240\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ framerate\=\(fraction\)49/2";
0:00:03.589155122     1      0x126ef70 DEBUG               GST_CAPS gstutils.c:2983:gst_pad_query_accept_caps:<capsfilter1:sink> accept caps of video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)2, profile=(string)high, codec_data=(buffer)01640014ffe1001d67640014acd94141fb016a0c020b4a000003000200000300651e28532c01000568ebecb22c, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)49/2
0:00:03.589167013     1      0x126ef70 DEBUG                  query gstquery.c:674:gst_query_new_custom: creating new query 0x7ff894c01370 accept-caps
0:00:03.589171008     1      0x126ef70 DEBUG               GST_PADS gstpad.c:3929:gst_pad_query:<capsfilter1:sink> doing query 0x7ff894c01370 (accept-caps)
0:00:03.589175793     1      0x126ef70 DEBUG             capsfilter gstcapsfilter.c:341:gst_capsfilter_accept_caps:<capsfilter1> can intersect: 0
0:00:03.589179594     1      0x126ef70 DEBUG               GST_PADS gstpad.c:3952:gst_pad_query:<capsfilter1:sink> sent query 0x7ff894c01370 (accept-caps), result 1
0:00:03.589183896     1      0x126ef70 DEBUG                default gstutils.c:2988:gst_pad_query_accept_caps:<capsfilter1:sink> query returned 0
0:00:03.589187404     1      0x126ef70 DEBUG               GST_CAPS gstpad.c:5475:pre_eventfunc_check:<capsfilter1:sink> caps video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)2, profile=(string)high, codec_data=(buffer)01640014ffe1001d67640014acd94141fb016a0c020b4a000003000200000300651e28532c01000568ebecb22c, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)49/2 not accepted
0:00:03.589198338     1      0x126ef70 DEBUG               GST_PADS gstpad.c:5718:gst_pad_send_event_unchecked:<capsfilter1:sink> pre event check failed
0:00:03.589201877     1      0x126ef70 DEBUG               GST_PADS gstpad.c:3839:push_sticky:<demux:video_0> result not-negotiated, mark pending events
0:00:03.589205822     1      0x126ef70 DEBUG             GST_MEMORY gstmemory.c:87:_gst_memory_free: free memory 0x7ff894bfdb90
0:00:03.589211112     1      0x126ef70 DEBUG                qtdemux qtdemux.c:6790:gst_qtdemux_process_adapter:<demux> Stopping, combined return flow not-negotiated