imxcompositer_g2d occasional severe video stuttering in pause/unpause

[Terry Barnaby]

Hi Nirbheek,

Many thanks for the reply and info. When we started the project quite a 
few of the IMX8MP hardware features were not supported by the FSL 
Freescale code and so we have based our system on the NXP BSP. I'm not 
sure that everything is supported now and I really don't want to be 
mixing packages and getting into horrible package version issues at this 
moment in time. However I might try that on a debug test release to see 
if it affects things.

We have to ship our first production systems in three weeks and 
everything has been tested in our system and is working well apart from 
this one issue and so we would like to try and debug and fix this item 
for now and can look at moving the whole Linux platform stack away from 
the NXP BSP for a later release.

Terry
On 09/11/2022 18:50, Nirbheek Chauhan wrote:
> Hi Terry,
>
> You seem to be using the NXP gstreamer fork, and the i.MX elements in
> that aren't that great. The recommended elements are in the Freescale
> gstreamer-imx repository:
>
> https://github.com/Freescale/gstreamer-imx/
>
> The compositor there is called "imxg2dcompositor", and there are
> equivalents for all other elements too. meta-freescale ships bitbake
> files for it:
>
> https://github.com/Freescale/meta-freescale/blob/kirkstone/recipes-multimedia/gstreamer/gstreamer1.0-plugins-imx_2.1.0.bb
>
> You will need to put this in your config (conf/local.conf or the OE
> distro config) to disable the NXP gstreamer fork entirely:
>
> PREFERRED_VERSION_gstreamer1.0:mx6-nxp-bsp = ""
> PREFERRED_VERSION_gstreamer1.0-plugins-base:mx6-nxp-bsp = ""
> PREFERRED_VERSION_gstreamer1.0-plugins-good:mx6-nxp-bsp = ""
> PREFERRED_VERSION_gstreamer1.0-plugins-bad:mx6-nxp-bsp = ""
> PREFERRED_VERSION_gstreamer1.0-libav:mx6-nxp-bsp = ""
> PREFERRED_VERSION_gstreamer1.0:mx8-nxp-bsp = ""
> PREFERRED_VERSION_gstreamer1.0-plugins-base:mx8-nxp-bsp = ""
> PREFERRED_VERSION_gstreamer1.0-plugins-good:mx8-nxp-bsp = ""
> PREFERRED_VERSION_gstreamer1.0-plugins-bad:mx8-nxp-bsp = ""
> PREFERRED_VERSION_gstreamer1.0-libav:mx8-nxp-bsp = ""
> PREFERRED_VERSION_gstreamer1.0 = ""
> PREFERRED_VERSION_gstreamer1.0-plugins-base = ""
> PREFERRED_VERSION_gstreamer1.0-plugins-good = ""
> PREFERRED_VERSION_gstreamer1.0-plugins-bad = ""
> PREFERRED_VERSION_gstreamer1.0-libav = ""
>
> If your OE-core layer has a very old gstreamer version, you will also
> need this layer: https://github.com/OSSystems/meta-gstreamer1.0 --
> pick the right branch in that repo for your yocto version.
>
> Basically, if you see a gstreamer package version with "imx" in it,
> like x.y.z.imx, that's the NXP fork and you need to fix up your
> config.
>
> Cheers,
> Nirbheek
>
> On Wed, Nov 9, 2022 at 8:16 PM Terry Barnaby via gstreamer-devel
> <gstreamer-devel at lists.freedesktop.org> wrote:
>> We are working on a video inspection system that is using an NXP IMX8MP
>> CPU based system that this using gstreamer for its image processing
>> pipeline. Generally the system is working fine but occasionally when you
>> pause and unpause the pipeline we get large video stuttering (frames
>> could take 20 seconds to update etc.).
>>
>> This is using NXP gstreamer Elements as well as NXP hardware blocks and
>> also one of our own gstreamer elements. I know this is not the standard
>> gstreamer code, but I am just after some debug advice as I have now
>> spent a week trying to get to the bottom of the issue and maybe someone
>> might have some ideas/pointers.
>>
>> I have got it down to a simple 'C' program example that shows the issue
>> after a few minutes. The basic pipeline for this is:
>>
>> imxcompositor_g2d name=c sink_0::alpha=1.0 sink_1::alpha=0.5 ! queue !
>> waylandsink v4l2src device=/dev/video3 ! beamdeinterlace !
>> video/x-raw,width=720,height=576,framerate=25/1 ! c.sink_0 appsrc
>> name=appsrc ! c.sink_1
>>
>> There is a 720x576 at 25 video stream from a CSI camera and a 720x576 at 25
>> overlay stream generated by an appsrc. The beamdeinterlace Element is a
>> very simple GstVideoFilterClass based Element that just uses a
>> transform_frame() function.
>>
>> If this stream is paused and unpaused, sometimes the camera video stream
>> will have this severe stuttering while the overlay stream is unaffected
>> when looking at the video output from the compositor.
>>
>> Adding probes to the imxcompositor_g2d sinks show that the buffer times
>> on both sinks are updating fine. Indeed if I tee of the stream after the
>> beamdeinterlace this looks fine whilst the issue is happening.
>>
>> Digging down into the imxcompositor_g2d it is using gstvideoaggregator
>> and gstaggregator. When I add printf's to its
>> gst_imxcompositor_aggregate_frames() function I see that its
>> gst_video_aggregator_pad_get_current_buffer() call is returning the same
>> frame continuously when the problem occurs. So it looks like something
>> in the GStreamer gstvideoaggregator or gstaggregator is dropping the
>> camera's video frames for some reason.
>>
>> Setting GST_DEBUG=imxcompositor:7 shows no obvious issues (no dropping
>> frame like messages).
>>
>> The GStreamer being used is 1.18.0 although I suspect with a few NXP mods.
>>
>> Has anyone seen this sort of behaviour with a compositor before ?
>>
>> Any ideas on what may be happening and where/how to look ?
>>