imxcompositer_g2d occasional severe video stuttering in pause/unpause

[Terry Barnaby]

We are working on a video inspection system that is using an NXP IMX8MP 
CPU based system that this using gstreamer for its image processing 
pipeline. Generally the system is working fine but occasionally when you 
pause and unpause the pipeline we get large video stuttering (frames 
could take 20 seconds to update etc.).

This is using NXP gstreamer Elements as well as NXP hardware blocks and 
also one of our own gstreamer elements. I know this is not the standard 
gstreamer code, but I am just after some debug advice as I have now 
spent a week trying to get to the bottom of the issue and maybe someone 
might have some ideas/pointers.

I have got it down to a simple 'C' program example that shows the issue 
after a few minutes. The basic pipeline for this is:

imxcompositor_g2d name=c sink_0::alpha=1.0 sink_1::alpha=0.5 ! queue ! 
waylandsink v4l2src device=/dev/video3 ! beamdeinterlace ! 
video/x-raw,width=720,height=576,framerate=25/1 ! c.sink_0 appsrc 
name=appsrc ! c.sink_1

There is a 720x576 at 25 video stream from a CSI camera and a 720x576 at 25 
overlay stream generated by an appsrc. The beamdeinterlace Element is a 
very simple GstVideoFilterClass based Element that just uses a 
transform_frame() function.

If this stream is paused and unpaused, sometimes the camera video stream 
will have this severe stuttering while the overlay stream is unaffected 
when looking at the video output from the compositor.

Adding probes to the imxcompositor_g2d sinks show that the buffer times 
on both sinks are updating fine. Indeed if I tee of the stream after the 
beamdeinterlace this looks fine whilst the issue is happening.

Digging down into the imxcompositor_g2d it is using gstvideoaggregator 
and gstaggregator. When I add printf's to its 
gst_imxcompositor_aggregate_frames() function I see that its 
gst_video_aggregator_pad_get_current_buffer() call is returning the same 
frame continuously when the problem occurs. So it looks like something 
in the GStreamer gstvideoaggregator or gstaggregator is dropping the 
camera's video frames for some reason.

Setting GST_DEBUG=imxcompositor:7 shows no obvious issues (no dropping 
frame like messages).

The GStreamer being used is 1.18.0 although I suspect with a few NXP mods.

Has anyone seen this sort of behaviour with a compositor before ?

Any ideas on what may be happening and where/how to look ?