Wrong PTS in AppSink and duplicate frames

[Nicolas Dufresne]

Le samedi 11 avril 2020 à 00:33 +0200, Robert Tari a écrit :
> Hello everyone,
> 
> I've been struggling with this for months now. I've looked in all the 
> GStreamer docs, read the whole Internet, but couldn't find a cure. I 
> would really be grateful for some help. Thanks in advance.
> 
> Here is what I'm trying to accomplish (simplified):
> 
> - seek to an exact decoded raw audio frame position
> - read a number of frames
> - (do some processing)
> - seek exactly past the previously read frames
> - read some more frames
> - (and so on)
> 
> What I'm expecting to see:
> - A contiguous buffer seamlessly memcpy'd together from AppSink's 
> buffers, that is basically a perfect copy of the decoded raw audio data. 
> This works OK for WAV files.
> 
> This is what happens instead:
> - For FLAC: after a seek, the first GstSample that I pull from AppSink 
> is always offset by exactly 1 frame, so I get a duplicate between the 
> end of the previous call to my read function and the current one
> - For various other formats: most GstSamples' GstBuffer->pts are either 
> -1, 0, or +1 frames of their expected position - so I either get 1 
> duplicate, get it right, or I'm 1 frame short
> 
> I've got the same results with hundreds of various test files, GStreamer 
> 1.14.5 and 1.16.2, on both 32 and 64bit versions of Linux.

It could be that the seeks are not accurate enough for you needs. Best
is to look in the appropriate parsers to find out if the seek is
effectively accurate for this codec or container. You may be able to
improve it, or not, since there exist formats that makes it impossible,
unless....

Unless you decode from the start and skip to the accurate position you
want. If that level of accuracy is required for you case, this might be
what you want to do. It's of course much slower.

> 
> 
> *****************************************************************************************
> 
> 
> Here is the pipeline:
> ---------------------
> 
> "filesrc location=FILE ! decodebin ! audioconvert ! audio/x-raw ! 
> appsink name=sink sync=FALSE"
> 
> 
> And here is my read function (simplified):
> ------------------------------------------
> 
> void read(gchar *lBuffer, guint64 nStartFrame, guint64 nFramesToRead, 
> GstAudioInfo *pAudioInfo, GstPipeline *pPipeline)
> {
>      guint64 nBytesToRead = nFramesToRead * pAudioInfo->bpf;
>      guint64 nBytesRead = 0;
>      // We need this temp buffer because of the offset thing
>      gchar *lBytes = g_malloc(nBytesToRead + 8192);
>      guint nOffsetBytes = 0;
> 
>      //We are paused here
> 
>      gint64 nTime = GST_FRAMES_TO_CLOCK_TIME(nStartFrame, 
> pAudioInfo->rate);
>      gst_element_seek_simple(GST_ELEMENT_CAST(pPipeline), 
> GST_FORMAT_TIME, GST_SEEK_FLAG_ACCURATE | GST_SEEK_FLAG_FLUSH, nTime);
>      gst_element_get_state(GST_ELEMENT_CAST(pPipeline), NULL, NULL, 
> GST_CLOCK_TIME_NONE);
> 
>      gst_element_query_position(GST_ELEMENT_CAST(pPipeline), 
> GST_FORMAT_TIME, &nTime);
> 
> 
>      // Very important to continue where we left off
>      g_assert(GST_CLOCK_TIME_TO_FRAMES(nTime, pAudioInfo->rate) == 
> nStartFrame);
> 
>      gst_element_set_state(GST_ELEMENT_CAST(pPipeline), 
> GST_STATE_PLAYING);
>      GstAppSink *pAppSink = 
> GST_APP_SINK_CAST(gst_bin_get_by_name(GST_BIN_CAST(pPipeline), "sink"));
> 
>      while (!gst_app_sink_is_eos(pAppSink))
>      {
>          GstSample *pSample = gst_app_sink_pull_sample(pAppSink);
> 
>          if (pSample)
>          {
>              GstBuffer *pBuffer = gst_sample_get_buffer(pSample);
>              gint64 nOfset = GST_CLOCK_TIME_TO_FRAMES(pBuffer->pts, 
> pAudioInfo->rate);
>              gint64 nWantOffset = (nBytesRead / pAudioInfo->bpf) + 
> nStartFrame;
> 
>              if (nOfset > nWantOffset)
>              {
>                  //WAV and FLAC never get here
>                  g_error("Bad pBuffer->pts");
>              }
>              else if (nOfset < nWantOffset && nOffsetBytes == 0)
>              {
>                  // FLAC reports multiple bad PTSs, but only the first 
> one must be corrected
>                  nOffsetBytes = (nWantOffset - nOfset) * pAudioInfo->bpf;
>              }
> 
>              GstMapInfo pMapInfo;
>              gboolean bSuccess = gst_buffer_map(pBuffer, &pMapInfo, 
> GST_MAP_READ);
> 
>              if (bSuccess)
>              {
>                  memcpy(lBytes + nBytesRead, pMapInfo.data, 
> pMapInfo.size);
>                  nBytesRead += pMapInfo.size;
>                  gst_buffer_unmap(pBuffer, &pMapInfo);
>              }
>              else
>              {
>                  g_error("gst_buffer_map failed");
>              }
> 
>              gst_sample_unref(pSample);
>          }
>          else
>          {
>              g_error("gst_app_sink_pull_sample failed");
>          }
> 
>          // Normally, it would be: if (nBytesRead == nBytesToRead)
>          if (nBytesRead >= nBytesToRead + nOffsetBytes)
>          {
>              break;
>          }
>      }
> 
>      gst_object_unref(pAppSink);
>      gst_element_set_state(GST_ELEMENT_CAST(pPipeline), 
> GST_STATE_PAUSED);
> 
>      memcpy(lBuffer, lBytes + nOffsetBytes, nBytesToRead);
>      g_free(lBytes);
> }
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel