imxcompositer_g2d occasional severe video stuttering in pause/unpause
Terry Barnaby
terry1 at beam.ltd.uk
Sun Nov 13 17:36:50 UTC 2022
Hi Nicolas,
Note, I am not asking for a fix just trying to get some help with
gstreamer in general to try and see if anyone else has seen a similar
issue and to get some pointers of where to look, as gstreamer/v4l2/NXP
drivers/kernel is quite complicated and I don't have much experience
with the gstreamer code base. I am digging into the code and will report
back to NXP and/or gstreamer if I find out what is causing this issue
for me.
As I'm getting nowhere with investigation, I thought I would try the
gstreamer1.0-plugins-imx plugin as suggested to see how that fares.
Unfortunately it didn't build using the NXP hardknott release I was
using so I have built the latest NXP Yokto kirkstone LF5.15.52_2.1.0 for
our board and have tried to build the gstreamer1.0-plugins-imx but there
are issues.
I am using the NXP Yocto repo install with all of the versions of things
that its repo pulls in. As suggested I have "disabled the NXP gstreamer
fork entirely" using then "PREFERRED_VERSION_gstreamer1.0*" settings in
my local.conf.
But when I build "bitbake gstreamer1.0-plugins-imx" I get an error when
it tries to build libimxvpuapi2. It looks like this is looking for
-lhantro_h1 which does not exist.
Is there any info on how to build the gstreamer1.0-plugins-imx over the
NXP Yokto LF5.15.52_2.1.0 BSP ?
Terry
On 12/11/2022 01:34, Nicolas Dufresne wrote:
>
>
> Le jeu. 10 nov. 2022, 03 h 17, Terry Barnaby via gstreamer-devel
> <gstreamer-devel at lists.freedesktop.org> a écrit :
>
> Hi Nirbheek,
>
> Many thanks for the reply and info. When we started the project
> quite a
> few of the IMX8MP hardware features were not supported by the FSL
> Freescale code and so we have based our system on the NXP BSP. I'm
> not
> sure that everything is supported now and I really don't want to be
> mixing packages and getting into horrible package version issues
> at this
> moment in time. However I might try that on a debug test release
> to see
> if it affects things.
>
> We have to ship our first production systems in three weeks and
> everything has been tested in our system and is working well apart
> from
> this one issue and so we would like to try and debug and fix this
> item
> for now and can look at moving the whole Linux platform stack away
> from
> the NXP BSP for a later release.
>
>
> Gstreamer-imx is meant to run over the BSP. Its a drop-in replacement
> usually. That being said, we do have folks helping with gstreamer-imx
> in GStreamer community, but the NXP plugins though are solely
> maintained by NXP. So for continuing to use these and asking for a
> fix, please use NXP support and/or user forum.
>
> Nicolas
>
>
>
> Terry
> On 09/11/2022 18:50, Nirbheek Chauhan wrote:
> > Hi Terry,
> >
> > You seem to be using the NXP gstreamer fork, and the i.MX
> elements in
> > that aren't that great. The recommended elements are in the
> Freescale
> > gstreamer-imx repository:
> >
> > https://github.com/Freescale/gstreamer-imx/
> >
> > The compositor there is called "imxg2dcompositor", and there are
> > equivalents for all other elements too. meta-freescale ships bitbake
> > files for it:
> >
> >
> https://github.com/Freescale/meta-freescale/blob/kirkstone/recipes-multimedia/gstreamer/gstreamer1.0-plugins-imx_2.1.0.bb
> >
> > You will need to put this in your config (conf/local.conf or the OE
> > distro config) to disable the NXP gstreamer fork entirely:
> >
> > PREFERRED_VERSION_gstreamer1.0:mx6-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-base:mx6-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-good:mx6-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-bad:mx6-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-libav:mx6-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0:mx8-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-base:mx8-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-good:mx8-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-bad:mx8-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0-libav:mx8-nxp-bsp = ""
> > PREFERRED_VERSION_gstreamer1.0 = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-base = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-good = ""
> > PREFERRED_VERSION_gstreamer1.0-plugins-bad = ""
> > PREFERRED_VERSION_gstreamer1.0-libav = ""
> >
> > If your OE-core layer has a very old gstreamer version, you will
> also
> > need this layer: https://github.com/OSSystems/meta-gstreamer1.0 --
> > pick the right branch in that repo for your yocto version.
> >
> > Basically, if you see a gstreamer package version with "imx" in it,
> > like x.y.z.imx, that's the NXP fork and you need to fix up your
> > config.
> >
> > Cheers,
> > Nirbheek
> >
> > On Wed, Nov 9, 2022 at 8:16 PM Terry Barnaby via gstreamer-devel
> > <gstreamer-devel at lists.freedesktop.org> wrote:
> >> We are working on a video inspection system that is using an
> NXP IMX8MP
> >> CPU based system that this using gstreamer for its image processing
> >> pipeline. Generally the system is working fine but occasionally
> when you
> >> pause and unpause the pipeline we get large video stuttering
> (frames
> >> could take 20 seconds to update etc.).
> >>
> >> This is using NXP gstreamer Elements as well as NXP hardware
> blocks and
> >> also one of our own gstreamer elements. I know this is not the
> standard
> >> gstreamer code, but I am just after some debug advice as I have now
> >> spent a week trying to get to the bottom of the issue and maybe
> someone
> >> might have some ideas/pointers.
> >>
> >> I have got it down to a simple 'C' program example that shows
> the issue
> >> after a few minutes. The basic pipeline for this is:
> >>
> >> imxcompositor_g2d name=c sink_0::alpha=1.0 sink_1::alpha=0.5 !
> queue !
> >> waylandsink v4l2src device=/dev/video3 ! beamdeinterlace !
> >> video/x-raw,width=720,height=576,framerate=25/1 ! c.sink_0 appsrc
> >> name=appsrc ! c.sink_1
> >>
> >> There is a 720x576 at 25 video stream from a CSI camera and a
> 720x576 at 25
> >> overlay stream generated by an appsrc. The beamdeinterlace
> Element is a
> >> very simple GstVideoFilterClass based Element that just uses a
> >> transform_frame() function.
> >>
> >> If this stream is paused and unpaused, sometimes the camera
> video stream
> >> will have this severe stuttering while the overlay stream is
> unaffected
> >> when looking at the video output from the compositor.
> >>
> >> Adding probes to the imxcompositor_g2d sinks show that the
> buffer times
> >> on both sinks are updating fine. Indeed if I tee of the stream
> after the
> >> beamdeinterlace this looks fine whilst the issue is happening.
> >>
> >> Digging down into the imxcompositor_g2d it is using
> gstvideoaggregator
> >> and gstaggregator. When I add printf's to its
> >> gst_imxcompositor_aggregate_frames() function I see that its
> >> gst_video_aggregator_pad_get_current_buffer() call is returning
> the same
> >> frame continuously when the problem occurs. So it looks like
> something
> >> in the GStreamer gstvideoaggregator or gstaggregator is
> dropping the
> >> camera's video frames for some reason.
> >>
> >> Setting GST_DEBUG=imxcompositor:7 shows no obvious issues (no
> dropping
> >> frame like messages).
> >>
> >> The GStreamer being used is 1.18.0 although I suspect with a
> few NXP mods.
> >>
> >> Has anyone seen this sort of behaviour with a compositor before ?
> >>
> >> Any ideas on what may be happening and where/how to look ?
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20221113/20948209/attachment.htm>
More information about the gstreamer-devel
mailing list