Using a queue element to store video frames

[Matthew Waters]

Hi,

On 11/11/21 04:16, Ranti Endeley via gstreamer-devel wrote:
> Hi,
>
> I am quite new to gstreamer. I am trying to develop a couple of plugins.
>
>   * one which examines video frames in a buffer alongside inference
>     metadata (object detection and classification) - this plugin then
>     emits a custom out of bounds event downstream which is acted upon
>     by the second plugin
>   * the second plugin is separated from the 1st by a queue (the idea
>     being to block the queue until the custom event notifies it to
>     start capturing the frames in the buffer)
>
> My idea is to use the queue (leaky=downstream) between the two plugins
> to store the buffers until an event is detected by the 1st plugin. I
> would like the queue to fill up with buffers and drop the oldest
> buffers until the downstream plugin is prepared to accept them. This
> should in theory then allow the 2nd plugin to capture the video frames
> emitted prior to the event occurring so that I can have a record of
> some seconds of video prior to the event which triggered the capture
> (by increasing the max-size-time property of the queue to match the
> amount of time I want to store before an event).
>

Yes, this is possible and doable as you describe.

> My pipeline is like this:
> video/x-raw + inference metadata -> videoconvert ! trigger plugin !
> queue max-size-buffers=G_MAXUINT, "max-size-bytes=G_MAXUINT,
> max-size-time=60000000000,leaky=downstream ! capture plugin !
> videoconvert ! videoscale ! videorate ! theoraenc ! oggmux ! filesink
> [hoping to store up to 60 seconds of videoframes]
>
> Testing the above pipeline using videotestsrc (no metadata yet, just
> passing through all frames) has brought up some issues that are a
> little hard for me to understand.
>
>   * The output video runs for much longer than expected (for example 2
>     seconds of runtime results in about 30 seconds of video)
>   * When the leaky=downstream option is set on the queue frames are
>     dropped much earlier than I would expect (leading to a very choppy
>     output video - which incidentally is still longer than expected).
>
> My questions:
>
>   * Is what I am trying to do possible with the pipeline I have
>     described above? If not, why and what am I missing?
>   * Why is the video length being output disproportionate to the run
>     time of the pipeline?
>

If you are running a source that just outputs data as fast as possible
(videotestsrc) with a sink element that doesn't not synchronise on the
clock (filesink), then you don't have anything 'realtime' here blocking
the execution based on the buffer timestamps.  The amount of resulting
data depends on how fast it can be generated.  Whether or not the queue
drops data depends on whether downstream processes data faster or slower
than upstream.

You can either use videotestsrc is-live=true or insert a clocksync
element at an approprate place before the leaky queue.

Cheers
-Matt

> Thanks in advance for your assistance.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20211115/f2ed0aac/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20211115/f2ed0aac/attachment.sig>