Accurately seeking to frames?

[Will McElderry]

Hi All,


TLDR version:

I am trying to accurately seek to a known frame in a video file (usually 
an MP4 with an h264 stream inside captured from an RTSP camera).

[1] Is Buffer.pts in Segment time, and need mapping to stream time with 
'Segment.to_stream_time(Buffer.pts)' ?

[2] Should Pipeline.seek( ... buffer_time ... ) mean the next frame I 
receive is the Buffer I saw at that stream time?

     (using a file src, pipeline does support seeking, details of '...'  
in seek function given below if useful)


Thanks!

Will.


_Verbose context:_

I am working with some legacy code and have been asked to make it seek 
accurately to a specific frame. However things are not working as I'd 
expect.

I'd love some feedback on my understanding, pointers to make things work 
or knowledgeable person to confirm if this could be an issue!


 From comments

here 
<https://gstreamer.freedesktop.org/documentation/application-development/advanced/clocks.html?gi-language=python#buffer-streamtime> 
(The buffer stream-time, ... [is] calculated from the buffer timestamps 
and the preceding |SEGMENT| event.)

and

here 
<https://gstreamer.freedesktop.org/documentation/gstreamer/gstsegment.html?gi-language=python#gst_segment_to_stream_time> 
(Gst.Segment.to_stream_time ... is typically used ... to operate on the 
stream time of the buffers it receives)

I believe Buffer.pts is in Segment time.  Is that correct?

(I cannot see where it explicitly stated anywhere what time it's in, so 
I could be easily wrong!)


Building on that understanding, I'm trying to obtain the *stream time* 
of a frame from a video that is provided to an appsink with 
'pull_preroll()' with the following method:

     seg = sample.get_segment()

     buf = sample.get_buffer()

     stream_time = seg.to_stream_time( buf.pts )


I then attempt to seek to a known frame's *stream time* I've obtained 
using the method above with:

     pipeline.seek(
         1.0, Gst.Format.TIME,

         Gst.SeekFlags.ACCURATE | Gst.SeekFlags.FLUSH,

         Gst.SeekType.SET, target_time,

         Gst.SeekType.NONE, 0
     )

    await_async_done()

However, I do not get the frame I started with (sometimes I do, but 
generally it's one or two out, usually prior to the frame I 
anticipate).  I've tried adding an 'epsilon' value to see if it is a 
rounding issue, but that doesn't seem to apply either.

Can anyone make any comments or recommendations?


Thanks!


Will.



P.S.

I've also tried the following based on seeking by Buffers, but it 
doesn't seem to work with my pipeline.  From docs I read I think this is 
not unexpected...

     pipeline.seek(
         1.0, Gst.Format.Buffers,

         Gst.SeekFlags.ACCURATE | Gst.SeekFlags.FLUSH,

         Gst.SeekType.SET, target_frame_index,

         Gst.SeekType.NONE, 0
     )
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20240112/df4ea50a/attachment.htm>