imxcompositer_g2d occasional severe video stuttering in pause/unpause

Sat Nov 12 01:34:20 UTC 2022

Le jeu. 10 nov. 2022, 03 h 17, Terry Barnaby via gstreamer-devel <
gstreamer-devel at lists.freedesktop.org> a écrit :

> Hi Nirbheek,
>
> Many thanks for the reply and info. When we started the project quite a
> few of the IMX8MP hardware features were not supported by the FSL
> Freescale code and so we have based our system on the NXP BSP. I'm not
> sure that everything is supported now and I really don't want to be
> mixing packages and getting into horrible package version issues at this
> moment in time. However I might try that on a debug test release to see
> if it affects things.
>
> We have to ship our first production systems in three weeks and
> everything has been tested in our system and is working well apart from
> this one issue and so we would like to try and debug and fix this item
> for now and can look at moving the whole Linux platform stack away from
> the NXP BSP for a later release.
>

Gstreamer-imx is meant to run over the BSP. Its a drop-in replacement
usually. That being said, we do have folks helping with gstreamer-imx in
GStreamer community, but the NXP plugins though are solely maintained by
NXP. So for continuing to use these and asking for a fix, please use NXP
support and/or user forum.

Nicolas

> Terry
> On 09/11/2022 18:50, Nirbheek Chauhan wrote:
> > Hi Terry,
> >
> > You seem to be using the NXP gstreamer fork, and the i.MX elements in
> > that aren't that great. The recommended elements are in the Freescale
> > gstreamer-imx repository:
> >
> > https://github.com/Freescale/gstreamer-imx/
> >
> > The compositor there is called "imxg2dcompositor", and there are
> > equivalents for all other elements too. meta-freescale ships bitbake
> > files for it:
> >
> >
> https://github.com/Freescale/meta-freescale/blob/kirkstone/recipes-multimedia/gstreamer/gstreamer1.0-plugins-imx_2.1.0.bb
> >
> > You will need to put this in your config (conf/local.conf or the OE
> > distro config) to disable the NXP gstreamer fork entirely:
> >
> > PREFERRED_VERSION_gstreamer1.0:mx6-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-base:mx6-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-good:mx6-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-bad:mx6-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-libav:mx6-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0:mx8-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-base:mx8-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-good:mx8-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-bad:mx8-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-libav:mx8-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0 = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-base = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-good = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-bad = ""
> > PREFERRED_VERSION_gstreamer1.0-libav = ""
> >
> > If your OE-core layer has a very old gstreamer version, you will also
> > need this layer: https://github.com/OSSystems/meta-gstreamer1.0 --
> > pick the right branch in that repo for your yocto version.
> >
> > Basically, if you see a gstreamer package version with "imx" in it,
> > like x.y.z.imx, that's the NXP fork and you need to fix up your
> > config.
> >
> > Cheers,
> > Nirbheek
> >
> > On Wed, Nov 9, 2022 at 8:16 PM Terry Barnaby via gstreamer-devel
> > <gstreamer-devel at lists.freedesktop.org> wrote:
> >> We are working on a video inspection system that is using an NXP IMX8MP
> >> CPU based system that this using gstreamer for its image processing
> >> pipeline. Generally the system is working fine but occasionally when you
> >> pause and unpause the pipeline we get large video stuttering (frames
> >> could take 20 seconds to update etc.).
> >>
> >> This is using NXP gstreamer Elements as well as NXP hardware blocks and
> >> also one of our own gstreamer elements. I know this is not the standard
> >> gstreamer code, but I am just after some debug advice as I have now
> >> spent a week trying to get to the bottom of the issue and maybe someone
> >> might have some ideas/pointers.
> >>
> >> I have got it down to a simple 'C' program example that shows the issue
> >> after a few minutes. The basic pipeline for this is:
> >>
> >> imxcompositor_g2d name=c sink_0::alpha=1.0 sink_1::alpha=0.5 ! queue !
> >> waylandsink v4l2src device=/dev/video3 ! beamdeinterlace !
> >> video/x-raw,width=720,height=576,framerate=25/1 ! c.sink_0 appsrc
> >> name=appsrc ! c.sink_1
> >>
> >> There is a 720x576 at 25 video stream from a CSI camera and a 720x576 at 25
> >> overlay stream generated by an appsrc. The beamdeinterlace Element is a
> >> very simple GstVideoFilterClass based Element that just uses a
> >> transform_frame() function.
> >>
> >> If this stream is paused and unpaused, sometimes the camera video stream
> >> will have this severe stuttering while the overlay stream is unaffected
> >> when looking at the video output from the compositor.
> >>
> >> Adding probes to the imxcompositor_g2d sinks show that the buffer times
> >> on both sinks are updating fine. Indeed if I tee of the stream after the
> >> beamdeinterlace this looks fine whilst the issue is happening.
> >>
> >> Digging down into the imxcompositor_g2d it is using gstvideoaggregator
> >> and gstaggregator. When I add printf's to its
> >> gst_imxcompositor_aggregate_frames() function I see that its
> >> gst_video_aggregator_pad_get_current_buffer() call is returning the same
> >> frame continuously when the problem occurs. So it looks like something
> >> in the GStreamer gstvideoaggregator or gstaggregator is dropping the
> >> camera's video frames for some reason.
> >>
> >> Setting GST_DEBUG=imxcompositor:7 shows no obvious issues (no dropping
> >> frame like messages).
> >>
> >> The GStreamer being used is 1.18.0 although I suspect with a few NXP
> mods.
> >>
> >> Has anyone seen this sort of behaviour with a compositor before ?
> >>
> >> Any ideas on what may be happening and where/how to look ?
> >>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20221111/2748b081/attachment.htm>