imxcompositer_g2d occasional severe video stuttering in pause/unpause

Mon Nov 14 17:49:02 UTC 2022

Hi Carlos,

No I am using: 
https://www.nxp.com/design/software/embedded-software/i-mx-software/embedded-linux-for-i-mx-applications-processors:IMXLINUX 

And am currently trying their latest Linux 5.15.52_2.1.0 kirkstone 
release with the hope that it will match the system that will allow the 
https://github.com/Freescale/gstreamer-imx/ gstreamer plugins to be used 
as a test. This appears to be using gstreamer1.0-1.20.0.imx-r0.armv8a by 
default.

Terry
On 14/11/2022 09:16, Carlos Rafael Giani via gstreamer-devel wrote:
>
> Hi Terry,
>
> you say that you are using the NXP Yocto repo install. Are you 
> referring to https://github.com/Freescale/fsl-community-bsp-platform ?
>
>
> On 13.11.22 18:36, Terry Barnaby via gstreamer-devel wrote:
>> Hi Nicolas,
>>
>> Note, I am not asking for a fix just trying to get some help with 
>> gstreamer in general to try and see if anyone else has seen a similar 
>> issue and to get some pointers of where to look, as 
>> gstreamer/v4l2/NXP drivers/kernel is quite complicated and I don't 
>> have much experience with the gstreamer code base. I am digging into 
>> the code and will report back to NXP and/or gstreamer if I find out 
>> what is causing this issue for me.
>>
>> As I'm getting nowhere with investigation, I thought I would try the 
>> gstreamer1.0-plugins-imx plugin as suggested to see how that fares. 
>> Unfortunately it didn't build using the NXP hardknott release I was 
>> using so I have built the latest NXP Yokto kirkstone LF5.15.52_2.1.0 
>> for our board and have tried to build the gstreamer1.0-plugins-imx 
>> but there are issues.
>>
>> I am using the NXP Yocto repo install with all of the versions of 
>> things that its repo pulls in. As suggested I have "disabled the NXP 
>> gstreamer fork entirely" using then "PREFERRED_VERSION_gstreamer1.0*" 
>> settings in my local.conf.
>> But when I build "bitbake gstreamer1.0-plugins-imx" I get an error 
>> when it tries to build libimxvpuapi2. It looks like this is looking 
>> for -lhantro_h1 which does not exist.
>>
>> Is there any info on how to build the gstreamer1.0-plugins-imx over 
>> the NXP Yokto LF5.15.52_2.1.0 BSP ?
>>
>>
>> Terry
>> On 12/11/2022 01:34, Nicolas Dufresne wrote:
>>>
>>>
>>> Le jeu. 10 nov. 2022, 03 h 17, Terry Barnaby via gstreamer-devel 
>>> <gstreamer-devel at lists.freedesktop.org> a écrit :
>>>
>>>     Hi Nirbheek,
>>>
>>>     Many thanks for the reply and info. When we started the project
>>>     quite a
>>>     few of the IMX8MP hardware features were not supported by the FSL
>>>     Freescale code and so we have based our system on the NXP BSP.
>>>     I'm not
>>>     sure that everything is supported now and I really don't want to be
>>>     mixing packages and getting into horrible package version issues
>>>     at this
>>>     moment in time. However I might try that on a debug test release
>>>     to see
>>>     if it affects things.
>>>
>>>     We have to ship our first production systems in three weeks and
>>>     everything has been tested in our system and is working well
>>>     apart from
>>>     this one issue and so we would like to try and debug and fix
>>>     this item
>>>     for now and can look at moving the whole Linux platform stack
>>>     away from
>>>     the NXP BSP for a later release.
>>>
>>>
>>> Gstreamer-imx is meant to run over the BSP. Its a drop-in 
>>> replacement usually. That being said, we do have folks helping with 
>>> gstreamer-imx in GStreamer community, but the NXP plugins though are 
>>> solely maintained by NXP. So for continuing to use these and asking 
>>> for a fix, please use NXP support and/or user forum.
>>>
>>> Nicolas
>>>
>>>
>>>
>>>     Terry
>>>     On 09/11/2022 18:50, Nirbheek Chauhan wrote:
>>>     > Hi Terry,
>>>     >
>>>     > You seem to be using the NXP gstreamer fork, and the i.MX
>>>     elements in
>>>     > that aren't that great. The recommended elements are in the
>>>     Freescale
>>>     > gstreamer-imx repository:
>>>     >
>>>     > https://github.com/Freescale/gstreamer-imx/
>>>     >
>>>     > The compositor there is called "imxg2dcompositor", and there are
>>>     > equivalents for all other elements too. meta-freescale ships
>>>     bitbake
>>>     > files for it:
>>>     >
>>>     >
>>>     https://github.com/Freescale/meta-freescale/blob/kirkstone/recipes-multimedia/gstreamer/gstreamer1.0-plugins-imx_2.1.0.bb
>>>     >
>>>     > You will need to put this in your config (conf/local.conf or
>>>     the OE
>>>     > distro config) to disable the NXP gstreamer fork entirely:
>>>     >
>>>     > PREFERRED_VERSION_gstreamer1.0:mx6-nxp-bsp = ""
>>>     > PREFERRED_VERSION_gstreamer1.0-plugins-base:mx6-nxp-bsp = ""
>>>     > PREFERRED_VERSION_gstreamer1.0-plugins-good:mx6-nxp-bsp = ""
>>>     > PREFERRED_VERSION_gstreamer1.0-plugins-bad:mx6-nxp-bsp = ""
>>>     > PREFERRED_VERSION_gstreamer1.0-libav:mx6-nxp-bsp = ""
>>>     > PREFERRED_VERSION_gstreamer1.0:mx8-nxp-bsp = ""
>>>     > PREFERRED_VERSION_gstreamer1.0-plugins-base:mx8-nxp-bsp = ""
>>>     > PREFERRED_VERSION_gstreamer1.0-plugins-good:mx8-nxp-bsp = ""
>>>     > PREFERRED_VERSION_gstreamer1.0-plugins-bad:mx8-nxp-bsp = ""
>>>     > PREFERRED_VERSION_gstreamer1.0-libav:mx8-nxp-bsp = ""
>>>     > PREFERRED_VERSION_gstreamer1.0 = ""
>>>     > PREFERRED_VERSION_gstreamer1.0-plugins-base = ""
>>>     > PREFERRED_VERSION_gstreamer1.0-plugins-good = ""
>>>     > PREFERRED_VERSION_gstreamer1.0-plugins-bad = ""
>>>     > PREFERRED_VERSION_gstreamer1.0-libav = ""
>>>     >
>>>     > If your OE-core layer has a very old gstreamer version, you
>>>     will also
>>>     > need this layer: https://github.com/OSSystems/meta-gstreamer1.0 --
>>>     > pick the right branch in that repo for your yocto version.
>>>     >
>>>     > Basically, if you see a gstreamer package version with "imx"
>>>     in it,
>>>     > like x.y.z.imx, that's the NXP fork and you need to fix up your
>>>     > config.
>>>     >
>>>     > Cheers,
>>>     > Nirbheek
>>>     >
>>>     > On Wed, Nov 9, 2022 at 8:16 PM Terry Barnaby via gstreamer-devel
>>>     > <gstreamer-devel at lists.freedesktop.org> wrote:
>>>     >> We are working on a video inspection system that is using an
>>>     NXP IMX8MP
>>>     >> CPU based system that this using gstreamer for its image
>>>     processing
>>>     >> pipeline. Generally the system is working fine but
>>>     occasionally when you
>>>     >> pause and unpause the pipeline we get large video stuttering
>>>     (frames
>>>     >> could take 20 seconds to update etc.).
>>>     >>
>>>     >> This is using NXP gstreamer Elements as well as NXP hardware
>>>     blocks and
>>>     >> also one of our own gstreamer elements. I know this is not
>>>     the standard
>>>     >> gstreamer code, but I am just after some debug advice as I
>>>     have now
>>>     >> spent a week trying to get to the bottom of the issue and
>>>     maybe someone
>>>     >> might have some ideas/pointers.
>>>     >>
>>>     >> I have got it down to a simple 'C' program example that shows
>>>     the issue
>>>     >> after a few minutes. The basic pipeline for this is:
>>>     >>
>>>     >> imxcompositor_g2d name=c sink_0::alpha=1.0 sink_1::alpha=0.5
>>>     ! queue !
>>>     >> waylandsink v4l2src device=/dev/video3 ! beamdeinterlace !
>>>     >> video/x-raw,width=720,height=576,framerate=25/1 ! c.sink_0 appsrc
>>>     >> name=appsrc ! c.sink_1
>>>     >>
>>>     >> There is a 720x576 at 25 video stream from a CSI camera and a
>>>     720x576 at 25
>>>     >> overlay stream generated by an appsrc. The beamdeinterlace
>>>     Element is a
>>>     >> very simple GstVideoFilterClass based Element that just uses a
>>>     >> transform_frame() function.
>>>     >>
>>>     >> If this stream is paused and unpaused, sometimes the camera
>>>     video stream
>>>     >> will have this severe stuttering while the overlay stream is
>>>     unaffected
>>>     >> when looking at the video output from the compositor.
>>>     >>
>>>     >> Adding probes to the imxcompositor_g2d sinks show that the
>>>     buffer times
>>>     >> on both sinks are updating fine. Indeed if I tee of the
>>>     stream after the
>>>     >> beamdeinterlace this looks fine whilst the issue is happening.
>>>     >>
>>>     >> Digging down into the imxcompositor_g2d it is using
>>>     gstvideoaggregator
>>>     >> and gstaggregator. When I add printf's to its
>>>     >> gst_imxcompositor_aggregate_frames() function I see that its
>>>     >> gst_video_aggregator_pad_get_current_buffer() call is
>>>     returning the same
>>>     >> frame continuously when the problem occurs. So it looks like
>>>     something
>>>     >> in the GStreamer gstvideoaggregator or gstaggregator is
>>>     dropping the
>>>     >> camera's video frames for some reason.
>>>     >>
>>>     >> Setting GST_DEBUG=imxcompositor:7 shows no obvious issues (no
>>>     dropping
>>>     >> frame like messages).
>>>     >>
>>>     >> The GStreamer being used is 1.18.0 although I suspect with a
>>>     few NXP mods.
>>>     >>
>>>     >> Has anyone seen this sort of behaviour with a compositor before ?
>>>     >>
>>>     >> Any ideas on what may be happening and where/how to look ?
>>>     >>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20221114/b7c4f119/attachment.htm>