imxcompositer_g2d occasional severe video stuttering in pause/unpause
terry1 at beam.ltd.uk
Wed Nov 9 13:40:57 UTC 2022
We are working on a video inspection system that is using an NXP IMX8MP
CPU based system that this using gstreamer for its image processing
pipeline. Generally the system is working fine but occasionally when you
pause and unpause the pipeline we get large video stuttering (frames
could take 20 seconds to update etc.).
This is using NXP gstreamer Elements as well as NXP hardware blocks and
also one of our own gstreamer elements. I know this is not the standard
gstreamer code, but I am just after some debug advice as I have now
spent a week trying to get to the bottom of the issue and maybe someone
might have some ideas/pointers.
I have got it down to a simple 'C' program example that shows the issue
after a few minutes. The basic pipeline for this is:
imxcompositor_g2d name=c sink_0::alpha=1.0 sink_1::alpha=0.5 ! queue !
waylandsink v4l2src device=/dev/video3 ! beamdeinterlace !
video/x-raw,width=720,height=576,framerate=25/1 ! c.sink_0 appsrc
name=appsrc ! c.sink_1
There is a 720x576 at 25 video stream from a CSI camera and a 720x576 at 25
overlay stream generated by an appsrc. The beamdeinterlace Element is a
very simple GstVideoFilterClass based Element that just uses a
If this stream is paused and unpaused, sometimes the camera video stream
will have this severe stuttering while the overlay stream is unaffected
when looking at the video output from the compositor.
Adding probes to the imxcompositor_g2d sinks show that the buffer times
on both sinks are updating fine. Indeed if I tee of the stream after the
beamdeinterlace this looks fine whilst the issue is happening.
Digging down into the imxcompositor_g2d it is using gstvideoaggregator
and gstaggregator. When I add printf's to its
gst_imxcompositor_aggregate_frames() function I see that its
gst_video_aggregator_pad_get_current_buffer() call is returning the same
frame continuously when the problem occurs. So it looks like something
in the GStreamer gstvideoaggregator or gstaggregator is dropping the
camera's video frames for some reason.
Setting GST_DEBUG=imxcompositor:7 shows no obvious issues (no dropping
frame like messages).
The GStreamer being used is 1.18.0 although I suspect with a few NXP mods.
Has anyone seen this sort of behaviour with a compositor before ?
Any ideas on what may be happening and where/how to look ?
More information about the gstreamer-devel