imxcompositer_g2d occasional severe video stuttering in pause/unpause
Carlos Rafael Giani
crg7475 at mailbox.org
Mon Nov 14 09:16:45 UTC 2022
Hi Terry,
you say that you are using the NXP Yocto repo install. Are you referring
to https://github.com/Freescale/fsl-community-bsp-platform ?
On 13.11.22 18:36, Terry Barnaby via gstreamer-devel wrote:
> Hi Nicolas,
>
> Note, I am not asking for a fix just trying to get some help with
> gstreamer in general to try and see if anyone else has seen a similar
> issue and to get some pointers of where to look, as gstreamer/v4l2/NXP
> drivers/kernel is quite complicated and I don't have much experience
> with the gstreamer code base. I am digging into the code and will
> report back to NXP and/or gstreamer if I find out what is causing this
> issue for me.
>
> As I'm getting nowhere with investigation, I thought I would try the
> gstreamer1.0-plugins-imx plugin as suggested to see how that fares.
> Unfortunately it didn't build using the NXP hardknott release I was
> using so I have built the latest NXP Yokto kirkstone LF5.15.52_2.1.0
> for our board and have tried to build the gstreamer1.0-plugins-imx but
> there are issues.
>
> I am using the NXP Yocto repo install with all of the versions of
> things that its repo pulls in. As suggested I have "disabled the NXP
> gstreamer fork entirely" using then "PREFERRED_VERSION_gstreamer1.0*"
> settings in my local.conf.
> But when I build "bitbake gstreamer1.0-plugins-imx" I get an error
> when it tries to build libimxvpuapi2. It looks like this is looking
> for -lhantro_h1 which does not exist.
>
> Is there any info on how to build the gstreamer1.0-plugins-imx over
> the NXP Yokto LF5.15.52_2.1.0 BSP ?
>
>
> Terry
> On 12/11/2022 01:34, Nicolas Dufresne wrote:
>>
>>
>> Le jeu. 10 nov. 2022, 03 h 17, Terry Barnaby via gstreamer-devel
>> <gstreamer-devel at lists.freedesktop.org> a écrit :
>>
>> Hi Nirbheek,
>>
>> Many thanks for the reply and info. When we started the project
>> quite a
>> few of the IMX8MP hardware features were not supported by the FSL
>> Freescale code and so we have based our system on the NXP BSP.
>> I'm not
>> sure that everything is supported now and I really don't want to be
>> mixing packages and getting into horrible package version issues
>> at this
>> moment in time. However I might try that on a debug test release
>> to see
>> if it affects things.
>>
>> We have to ship our first production systems in three weeks and
>> everything has been tested in our system and is working well
>> apart from
>> this one issue and so we would like to try and debug and fix this
>> item
>> for now and can look at moving the whole Linux platform stack
>> away from
>> the NXP BSP for a later release.
>>
>>
>> Gstreamer-imx is meant to run over the BSP. Its a drop-in replacement
>> usually. That being said, we do have folks helping with gstreamer-imx
>> in GStreamer community, but the NXP plugins though are solely
>> maintained by NXP. So for continuing to use these and asking for a
>> fix, please use NXP support and/or user forum.
>>
>> Nicolas
>>
>>
>>
>> Terry
>> On 09/11/2022 18:50, Nirbheek Chauhan wrote:
>> > Hi Terry,
>> >
>> > You seem to be using the NXP gstreamer fork, and the i.MX
>> elements in
>> > that aren't that great. The recommended elements are in the
>> Freescale
>> > gstreamer-imx repository:
>> >
>> > https://github.com/Freescale/gstreamer-imx/
>> >
>> > The compositor there is called "imxg2dcompositor", and there are
>> > equivalents for all other elements too. meta-freescale ships
>> bitbake
>> > files for it:
>> >
>> >
>> https://github.com/Freescale/meta-freescale/blob/kirkstone/recipes-multimedia/gstreamer/gstreamer1.0-plugins-imx_2.1.0.bb
>> >
>> > You will need to put this in your config (conf/local.conf or the OE
>> > distro config) to disable the NXP gstreamer fork entirely:
>> >
>> > PREFERRED_VERSION_gstreamer1.0:mx6-nxp-bsp = ""
>> > PREFERRED_VERSION_gstreamer1.0-plugins-base:mx6-nxp-bsp = ""
>> > PREFERRED_VERSION_gstreamer1.0-plugins-good:mx6-nxp-bsp = ""
>> > PREFERRED_VERSION_gstreamer1.0-plugins-bad:mx6-nxp-bsp = ""
>> > PREFERRED_VERSION_gstreamer1.0-libav:mx6-nxp-bsp = ""
>> > PREFERRED_VERSION_gstreamer1.0:mx8-nxp-bsp = ""
>> > PREFERRED_VERSION_gstreamer1.0-plugins-base:mx8-nxp-bsp = ""
>> > PREFERRED_VERSION_gstreamer1.0-plugins-good:mx8-nxp-bsp = ""
>> > PREFERRED_VERSION_gstreamer1.0-plugins-bad:mx8-nxp-bsp = ""
>> > PREFERRED_VERSION_gstreamer1.0-libav:mx8-nxp-bsp = ""
>> > PREFERRED_VERSION_gstreamer1.0 = ""
>> > PREFERRED_VERSION_gstreamer1.0-plugins-base = ""
>> > PREFERRED_VERSION_gstreamer1.0-plugins-good = ""
>> > PREFERRED_VERSION_gstreamer1.0-plugins-bad = ""
>> > PREFERRED_VERSION_gstreamer1.0-libav = ""
>> >
>> > If your OE-core layer has a very old gstreamer version, you
>> will also
>> > need this layer: https://github.com/OSSystems/meta-gstreamer1.0 --
>> > pick the right branch in that repo for your yocto version.
>> >
>> > Basically, if you see a gstreamer package version with "imx" in it,
>> > like x.y.z.imx, that's the NXP fork and you need to fix up your
>> > config.
>> >
>> > Cheers,
>> > Nirbheek
>> >
>> > On Wed, Nov 9, 2022 at 8:16 PM Terry Barnaby via gstreamer-devel
>> > <gstreamer-devel at lists.freedesktop.org> wrote:
>> >> We are working on a video inspection system that is using an
>> NXP IMX8MP
>> >> CPU based system that this using gstreamer for its image
>> processing
>> >> pipeline. Generally the system is working fine but
>> occasionally when you
>> >> pause and unpause the pipeline we get large video stuttering
>> (frames
>> >> could take 20 seconds to update etc.).
>> >>
>> >> This is using NXP gstreamer Elements as well as NXP hardware
>> blocks and
>> >> also one of our own gstreamer elements. I know this is not the
>> standard
>> >> gstreamer code, but I am just after some debug advice as I
>> have now
>> >> spent a week trying to get to the bottom of the issue and
>> maybe someone
>> >> might have some ideas/pointers.
>> >>
>> >> I have got it down to a simple 'C' program example that shows
>> the issue
>> >> after a few minutes. The basic pipeline for this is:
>> >>
>> >> imxcompositor_g2d name=c sink_0::alpha=1.0 sink_1::alpha=0.5 !
>> queue !
>> >> waylandsink v4l2src device=/dev/video3 ! beamdeinterlace !
>> >> video/x-raw,width=720,height=576,framerate=25/1 ! c.sink_0 appsrc
>> >> name=appsrc ! c.sink_1
>> >>
>> >> There is a 720x576 at 25 video stream from a CSI camera and a
>> 720x576 at 25
>> >> overlay stream generated by an appsrc. The beamdeinterlace
>> Element is a
>> >> very simple GstVideoFilterClass based Element that just uses a
>> >> transform_frame() function.
>> >>
>> >> If this stream is paused and unpaused, sometimes the camera
>> video stream
>> >> will have this severe stuttering while the overlay stream is
>> unaffected
>> >> when looking at the video output from the compositor.
>> >>
>> >> Adding probes to the imxcompositor_g2d sinks show that the
>> buffer times
>> >> on both sinks are updating fine. Indeed if I tee of the stream
>> after the
>> >> beamdeinterlace this looks fine whilst the issue is happening.
>> >>
>> >> Digging down into the imxcompositor_g2d it is using
>> gstvideoaggregator
>> >> and gstaggregator. When I add printf's to its
>> >> gst_imxcompositor_aggregate_frames() function I see that its
>> >> gst_video_aggregator_pad_get_current_buffer() call is
>> returning the same
>> >> frame continuously when the problem occurs. So it looks like
>> something
>> >> in the GStreamer gstvideoaggregator or gstaggregator is
>> dropping the
>> >> camera's video frames for some reason.
>> >>
>> >> Setting GST_DEBUG=imxcompositor:7 shows no obvious issues (no
>> dropping
>> >> frame like messages).
>> >>
>> >> The GStreamer being used is 1.18.0 although I suspect with a
>> few NXP mods.
>> >>
>> >> Has anyone seen this sort of behaviour with a compositor before ?
>> >>
>> >> Any ideas on what may be happening and where/how to look ?
>> >>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20221114/6c7d472d/attachment-0001.htm>
More information about the gstreamer-devel
mailing list