imxcompositer_g2d occasional severe video stuttering in pause/unpause

[Nirbheek Chauhan]

Hi Terry,

You seem to be using the NXP gstreamer fork, and the i.MX elements in
that aren't that great. The recommended elements are in the Freescale
gstreamer-imx repository:

https://github.com/Freescale/gstreamer-imx/

The compositor there is called "imxg2dcompositor", and there are
equivalents for all other elements too. meta-freescale ships bitbake
files for it:

https://github.com/Freescale/meta-freescale/blob/kirkstone/recipes-multimedia/gstreamer/gstreamer1.0-plugins-imx_2.1.0.bb

You will need to put this in your config (conf/local.conf or the OE
distro config) to disable the NXP gstreamer fork entirely:

PREFERRED_VERSION_gstreamer1.0:mx6-nxp-bsp = ""
PREFERRED_VERSION_gstreamer1.0-plugins-base:mx6-nxp-bsp = ""
PREFERRED_VERSION_gstreamer1.0-plugins-good:mx6-nxp-bsp = ""
PREFERRED_VERSION_gstreamer1.0-plugins-bad:mx6-nxp-bsp = ""
PREFERRED_VERSION_gstreamer1.0-libav:mx6-nxp-bsp = ""
PREFERRED_VERSION_gstreamer1.0:mx8-nxp-bsp = ""
PREFERRED_VERSION_gstreamer1.0-plugins-base:mx8-nxp-bsp = ""
PREFERRED_VERSION_gstreamer1.0-plugins-good:mx8-nxp-bsp = ""
PREFERRED_VERSION_gstreamer1.0-plugins-bad:mx8-nxp-bsp = ""
PREFERRED_VERSION_gstreamer1.0-libav:mx8-nxp-bsp = ""
PREFERRED_VERSION_gstreamer1.0 = ""
PREFERRED_VERSION_gstreamer1.0-plugins-base = ""
PREFERRED_VERSION_gstreamer1.0-plugins-good = ""
PREFERRED_VERSION_gstreamer1.0-plugins-bad = ""
PREFERRED_VERSION_gstreamer1.0-libav = ""

If your OE-core layer has a very old gstreamer version, you will also
need this layer: https://github.com/OSSystems/meta-gstreamer1.0 --
pick the right branch in that repo for your yocto version.

Basically, if you see a gstreamer package version with "imx" in it,
like x.y.z.imx, that's the NXP fork and you need to fix up your
config.

Cheers,
Nirbheek

On Wed, Nov 9, 2022 at 8:16 PM Terry Barnaby via gstreamer-devel
<gstreamer-devel at lists.freedesktop.org> wrote:
>
> We are working on a video inspection system that is using an NXP IMX8MP
> CPU based system that this using gstreamer for its image processing
> pipeline. Generally the system is working fine but occasionally when you
> pause and unpause the pipeline we get large video stuttering (frames
> could take 20 seconds to update etc.).
>
> This is using NXP gstreamer Elements as well as NXP hardware blocks and
> also one of our own gstreamer elements. I know this is not the standard
> gstreamer code, but I am just after some debug advice as I have now
> spent a week trying to get to the bottom of the issue and maybe someone
> might have some ideas/pointers.
>
> I have got it down to a simple 'C' program example that shows the issue
> after a few minutes. The basic pipeline for this is:
>
> imxcompositor_g2d name=c sink_0::alpha=1.0 sink_1::alpha=0.5 ! queue !
> waylandsink v4l2src device=/dev/video3 ! beamdeinterlace !
> video/x-raw,width=720,height=576,framerate=25/1 ! c.sink_0 appsrc
> name=appsrc ! c.sink_1
>
> There is a 720x576 at 25 video stream from a CSI camera and a 720x576 at 25
> overlay stream generated by an appsrc. The beamdeinterlace Element is a
> very simple GstVideoFilterClass based Element that just uses a
> transform_frame() function.
>
> If this stream is paused and unpaused, sometimes the camera video stream
> will have this severe stuttering while the overlay stream is unaffected
> when looking at the video output from the compositor.
>
> Adding probes to the imxcompositor_g2d sinks show that the buffer times
> on both sinks are updating fine. Indeed if I tee of the stream after the
> beamdeinterlace this looks fine whilst the issue is happening.
>
> Digging down into the imxcompositor_g2d it is using gstvideoaggregator
> and gstaggregator. When I add printf's to its
> gst_imxcompositor_aggregate_frames() function I see that its
> gst_video_aggregator_pad_get_current_buffer() call is returning the same
> frame continuously when the problem occurs. So it looks like something
> in the GStreamer gstvideoaggregator or gstaggregator is dropping the
> camera's video frames for some reason.
>
> Setting GST_DEBUG=imxcompositor:7 shows no obvious issues (no dropping
> frame like messages).
>
> The GStreamer being used is 1.18.0 although I suspect with a few NXP mods.
>
> Has anyone seen this sort of behaviour with a compositor before ?
>
> Any ideas on what may be happening and where/how to look ?
>