v4l2: No handling of no-show frames in decoder plugin

[Nicolas Dufresne]

Hi Patrik,

Le mer. 26 févr. 2025, 06 h 30, Pratik Pachange (QUIC) via gstreamer-devel <
gstreamer-devel at lists.freedesktop.org> a écrit :

>
>
> Hi Team,
>
>
>
> For AV1 codec format, there are certain no-show frames (show_frame flag
> set to 0) given by the video driver/fw as decoded output which needs to
> dropped/discarded by the client.
> On v4l2 side, the only way to identify such buffers is that their
> bytesused is 0 (or filledlen is 0).
>

The only AV1 decoder we have in upstream GStreamer is the stateless
decoder, which does not have this issue. I know some HW vendor added AV1 in
downstream kernel, without ever upstreaming it and getting their design
reviewed. This obviously lead to mistakes like this.

That being said, I'm not against fixing it with your suggestions. Though
some more thought needs to placed into that since that clashes with the
legacy way to signal EOS, and we also need to make sure it aligns with the
rest of V4L specific mainline.

An alternative is to define V4L2 AV1 as TU aligned. Then multiple frame
gets submitted to the decoder, and only one is to be produced. The downside
is that bitstream buffer size estimation will fall appart, since we have no
idea how many frames a bitstream (OUTPUT queue buffer) may endup.

Since this is a well know problem surface, we could also consider adding
signalling in the av1parse element.

>
> Currently in v4l2 plugins, there is no handling to drop such buffers.
> Can we plan to add any handling to drop/discard empty buffers (bytesused
> 0).
>

We do have some code to drop very old GstVideoFrame objects, but it takes
quite some time. It was added to workaround similar issues when with
interlaced video (H.264) mostly. Assuming it works with H.264 interlaced,
and does not go against the spec, I'll be happy to review and merge your
patches. Use the following to see the contribution manual.

https://gstreamer.freedesktop.org/documentation/contribute/index.html

Nicolas

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20250227/7235cc47/attachment.htm>