Accurately seeking to frames?

[Nicolas Dufresne]

Le vendredi 12 janvier 2024 à 16:50 +0000, Will McElderry via gstreamer-devel a
écrit :
>  
> Hi All,
>  
> 
>  
>  
> TLDR version:
>  
> I am trying to accurately seek to a known frame in a video file (usually an
> MP4 with an h264 stream inside captured from an RTSP camera).
This can only be performed if the source media have a fixed frame rate, just be
aware.
>  
> [1] Is Buffer.pts in Segment time, and need mapping to stream time with
> 'Segment.to_stream_time(Buffer.pts)' ?

Yes.

>  
> [2] Should Pipeline.seek( ... buffer_time ... ) mean the next frame I receive
> is the Buffer I saw at that stream time?
>  
>     (using a file src, pipeline does support seeking, details of '...'  in
> seek function given below if useful)

Seek time is always in stream time. You'll have to use the seeking flag
"GST_SEEK_FLAG_ACCURATE" to obtain the exact frame. In this context, demuxers
will make extra effort to ensure the first frame match as closely as possible
the requested stream time.

Nicolas

>  
> 
>  
> Thanks!
>  
> Will.
>  
> 
>  
>  
> Verbose context:
>  
>  
>  
> I am working with some legacy code and have been asked to make it seek
> accurately to a specific frame. However things are not working as I'd expect.
>  
> I'd love some feedback on my understanding, pointers to make things work or
> knowledgeable person to confirm if this could be an issue!
>  
>  
> From comments 
>  
>     here    (The buffer stream-time, ... [is] calculated from the buffer
> timestamps and the preceding SEGMENT event.)
>  
> and
>  
>     here    (Gst.Segment.to_stream_time ... is typically used ... to operate
> on the stream time of the buffers it receives)
>  
> I believe Buffer.pts is in Segment time.  Is that correct?
>  
> (I cannot see where it explicitly stated anywhere what time it's in, so I
> could be easily wrong!)
>  
> 
>  
>  
> Building on that understanding, I'm trying to obtain the stream time of a
> frame from a video that is provided to an appsink with 'pull_preroll()' with
> the following method:
>  
>     seg = sample.get_segment()
>  
>     buf = sample.get_buffer()
>  
>     stream_time = seg.to_stream_time( buf.pts )
>  
> 
>  
>  
> I then attempt to seek to a known frame's stream time I've obtained using the
> method above with:
>  
>     pipeline.seek(
>         1.0, Gst.Format.TIME,
>  
>         Gst.SeekFlags.ACCURATE | Gst.SeekFlags.FLUSH,
>  
>         Gst.SeekType.SET, target_time,
>  
>         Gst.SeekType.NONE, 0
>     )
>  
>    await_async_done()
>  
> However, I do not get the frame I started with (sometimes I do, but generally
> it's one or two out, usually prior to the frame I anticipate).  I've tried
> adding an 'epsilon' value to see if it is a rounding issue, but that doesn't
> seem to apply either.
>  
> Can anyone make any comments or recommendations?
Seeking to get a specific frame without some Format.FRAME (in theory DEFAULT for
video should be frame, but this is largely unimplemented since we don't track
frame numbers, and most container don't do that either). A slower, but more
accurate alternative is to use step operations (see gst_event_new_step()).
>  
> 
>  
>  
> Thanks!
>  
> 
>  
>  
> Will.
>  
> 
>  
>  
> 
>  
>  
> P.S.
>  
> I've also tried the following based on seeking by Buffers, but it doesn't seem
> to work with my pipeline.  From docs I read I think this is not unexpected...
>  
>     pipeline.seek(
>         1.0, Gst.Format.Buffers,
>  
>         Gst.SeekFlags.ACCURATE | Gst.SeekFlags.FLUSH,
>  
>         Gst.SeekType.SET, target_frame_index,
>  
>         Gst.SeekType.NONE, 0
>     )
>  
>  

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20240112/659888be/attachment.htm>