Wrong PTS in AppSink and duplicate frames
Robert Tari
robert at tari.in
Sun Apr 12 12:25:19 UTC 2020
Thanks for the answer.
If you look at my code sample you can see I have this check:
g_assert(GST_CLOCK_TIME_TO_FRAMES(nTime, pAudioInfo->rate) ==
nStartFrame);
It never fails. If it was the first buffer after the seek that has the
wrong PTS, I could simply offset my temp buffer (as I do with FLAC right
now), and read the rest. The problem is after the seek: when repeatedly
pulling AppSink buffers, they do not always follow the previous pull. If
I read the complete file from position 0 into the AppSink, I have the
same issue: some AppSink buffers do not start where previously pulled
buffer ended.
By the way, I tried replacing AppSink with GStreamSink and reading whole
files into a buffer: the output is 100% identical to the AppSink method:
a few frames are duplicated, and a few are overlapped.
Maybe there is an element to add to the pipeline, or a property that
could control this behaviour?
On 2020-04-12 03:36, Nicolas Dufresne wrote:
> Le samedi 11 avril 2020 à 00:33 +0200, Robert Tari a écrit :
>> Hello everyone,
>>
>> I've been struggling with this for months now. I've looked in all the
>> GStreamer docs, read the whole Internet, but couldn't find a cure. I
>> would really be grateful for some help. Thanks in advance.
>>
>> Here is what I'm trying to accomplish (simplified):
>>
>> - seek to an exact decoded raw audio frame position
>> - read a number of frames
>> - (do some processing)
>> - seek exactly past the previously read frames
>> - read some more frames
>> - (and so on)
>>
>> What I'm expecting to see:
>> - A contiguous buffer seamlessly memcpy'd together from AppSink's
>> buffers, that is basically a perfect copy of the decoded raw audio
>> data.
>> This works OK for WAV files.
>>
>> This is what happens instead:
>> - For FLAC: after a seek, the first GstSample that I pull from AppSink
>> is always offset by exactly 1 frame, so I get a duplicate between the
>> end of the previous call to my read function and the current one
>> - For various other formats: most GstSamples' GstBuffer->pts are
>> either
>> -1, 0, or +1 frames of their expected position - so I either get 1
>> duplicate, get it right, or I'm 1 frame short
>>
>> I've got the same results with hundreds of various test files,
>> GStreamer
>> 1.14.5 and 1.16.2, on both 32 and 64bit versions of Linux.
>
> It could be that the seeks are not accurate enough for you needs. Best
> is to look in the appropriate parsers to find out if the seek is
> effectively accurate for this codec or container. You may be able to
> improve it, or not, since there exist formats that makes it impossible,
> unless....
>
> Unless you decode from the start and skip to the accurate position you
> want. If that level of accuracy is required for you case, this might be
> what you want to do. It's of course much slower.
>
>>
>>
>> *****************************************************************************************
>>
>>
>> Here is the pipeline:
>> ---------------------
>>
>> "filesrc location=FILE ! decodebin ! audioconvert ! audio/x-raw !
>> appsink name=sink sync=FALSE"
>>
>>
>> And here is my read function (simplified):
>> ------------------------------------------
>>
>> void read(gchar *lBuffer, guint64 nStartFrame, guint64 nFramesToRead,
>> GstAudioInfo *pAudioInfo, GstPipeline *pPipeline)
>> {
>> guint64 nBytesToRead = nFramesToRead * pAudioInfo->bpf;
>> guint64 nBytesRead = 0;
>> // We need this temp buffer because of the offset thing
>> gchar *lBytes = g_malloc(nBytesToRead + 8192);
>> guint nOffsetBytes = 0;
>>
>> //We are paused here
>>
>> gint64 nTime = GST_FRAMES_TO_CLOCK_TIME(nStartFrame,
>> pAudioInfo->rate);
>> gst_element_seek_simple(GST_ELEMENT_CAST(pPipeline),
>> GST_FORMAT_TIME, GST_SEEK_FLAG_ACCURATE | GST_SEEK_FLAG_FLUSH, nTime);
>> gst_element_get_state(GST_ELEMENT_CAST(pPipeline), NULL, NULL,
>> GST_CLOCK_TIME_NONE);
>>
>> gst_element_query_position(GST_ELEMENT_CAST(pPipeline),
>> GST_FORMAT_TIME, &nTime);
>>
>>
>> // Very important to continue where we left off
>> g_assert(GST_CLOCK_TIME_TO_FRAMES(nTime, pAudioInfo->rate) ==
>> nStartFrame);
>>
>> gst_element_set_state(GST_ELEMENT_CAST(pPipeline),
>> GST_STATE_PLAYING);
>> GstAppSink *pAppSink =
>> GST_APP_SINK_CAST(gst_bin_get_by_name(GST_BIN_CAST(pPipeline),
>> "sink"));
>>
>> while (!gst_app_sink_is_eos(pAppSink))
>> {
>> GstSample *pSample = gst_app_sink_pull_sample(pAppSink);
>>
>> if (pSample)
>> {
>> GstBuffer *pBuffer = gst_sample_get_buffer(pSample);
>> gint64 nOfset = GST_CLOCK_TIME_TO_FRAMES(pBuffer->pts,
>> pAudioInfo->rate);
>> gint64 nWantOffset = (nBytesRead / pAudioInfo->bpf) +
>> nStartFrame;
>>
>> if (nOfset > nWantOffset)
>> {
>> //WAV and FLAC never get here
>> g_error("Bad pBuffer->pts");
>> }
>> else if (nOfset < nWantOffset && nOffsetBytes == 0)
>> {
>> // FLAC reports multiple bad PTSs, but only the first
>> one must be corrected
>> nOffsetBytes = (nWantOffset - nOfset) *
>> pAudioInfo->bpf;
>> }
>>
>> GstMapInfo pMapInfo;
>> gboolean bSuccess = gst_buffer_map(pBuffer, &pMapInfo,
>> GST_MAP_READ);
>>
>> if (bSuccess)
>> {
>> memcpy(lBytes + nBytesRead, pMapInfo.data,
>> pMapInfo.size);
>> nBytesRead += pMapInfo.size;
>> gst_buffer_unmap(pBuffer, &pMapInfo);
>> }
>> else
>> {
>> g_error("gst_buffer_map failed");
>> }
>>
>> gst_sample_unref(pSample);
>> }
>> else
>> {
>> g_error("gst_app_sink_pull_sample failed");
>> }
>>
>> // Normally, it would be: if (nBytesRead == nBytesToRead)
>> if (nBytesRead >= nBytesToRead + nOffsetBytes)
>> {
>> break;
>> }
>> }
>>
>> gst_object_unref(pAppSink);
>> gst_element_set_state(GST_ELEMENT_CAST(pPipeline),
>> GST_STATE_PAUSED);
>>
>> memcpy(lBuffer, lBytes + nOffsetBytes, nBytesToRead);
>> g_free(lBytes);
>> }
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
More information about the gstreamer-devel
mailing list