H264 stream without vui-parameters not handled correctly any more

[gstreamer at iktek.de]

Hi together,
Again I'm experiencing a regression between gstreamer 1.16.3 on yocto dunfell with kernel 5.4.149 refering as the "old setup" and gstreamer 1.18.4 on yocto hardknott with kernel 5.10.78 refering as
the "new
setup" and am wondering where to dig at it:

I'm creating a h264 stream on another device with help of v4l2h264enc with:

gst-launch-1.0 -vvvv v4l2src device=/dev/video/chts_camera_jpeg ! image/jpeg,width=800,height=600,framerate=15/1 ! v4l2jpegdec output-io-mode=dmabuf-import ! v4l2convert output-io-mode=dmabuf-import !
video/x-raw,co
lorimetry=bt709 ! v4l2h264enc output-io-mode=dmabuf-import ! 'video/x-h264,level=(string)3.2' ! rtph264pay ! udpsink port=5000 host=192.168.30.114

This one is accepted with the "old" setup and hardware decoding using:

gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
videoconvert ! kmssink

and also with the old and the new setup with software decoding using:

gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! avdec_h264 ! kmssink

-----------------------------------------------------------------------------------------

After patching the kernel with the Patch at "[PATCH v3 1/2] media: coda: Fix reported H264 profile", the stream's caps are accepted, but the stream does not start to run.
I already found out that the v4lh264enc in combination with coda and imx-hardware does NOT insert vui-parameters in the stream.
So h264parse will configure the framerate to 0/1.

The above decoding pipeline will stop at the first still picture. This was also working with the "old setup".
When simply setting "sync=false" on kmssink the pipeline starts to run, but there is a increasing latency observed because the pictures will flow in with 15 fps but kmssink will only process them with
about 11 fps, so qeueus will run full.

I also can avoid this with adding a rate-converter in between, but this solution doesn't look to me as it should be that way.

Changing the pipeline for using avdec_h264 the problem doesn't exist.

For my understanding vui-parameters are not mandatory and there may be a bunch of encoders who do not insert them?
I also have found other people who have similar problems but without a solution at: https://community.nxp.com/t5/i-MX-Processors/vpuenc-framerate-data-missing-vui-parameters-timing-info/m-p/310478.

When I play content from my pc (x264enc) to that pipelines (with vui-parameters set) all pipelines are working gracefully.

-------------------------------------------------------------------------------------------

SUMMARY OF PIPELINES:

NOT RUNNING (still picture, time stops at 00:00:00)
gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
v4l2convert ! kmssink

RUNNING but increasing latency:
gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
v4l2convert ! kmssink sync = false

RUNNING but "hacky", guessed framerate of kmssink
gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
v4l2convert ! queue ! videorate ! "video/x-raw, framerate=10/1" ! kmssink sync = false

RUNNING but it's a fakesink:
gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
v4l2convert ! fakesink

RUNNING but sw-decoding:
gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! avdec_h264 !
v4l2convert ! kmssink


My questions:
Is that vui-information a mandatory requirement for IP-communications standards and may there be a way to insert them with help of other gstreamer-elements "post-encoding".

If vui-information is not mandatory we would IMHO need to make sure that a "default" gstreamer pipeline without hacky parameters could handle that gracefully.
Does anybody have an idea what commit could have created that regression so that kmssink stops running now?

Thanks in advance.
Best Regards
Pascal Speck