H264 stream without vui-parameters not handled correctly any more

[Nicolas Dufresne]

Hi,

Le jeudi 10 mars 2022 à 12:06 +0100, gstreamer at iktek.de a écrit :
> Hi together,
> Again I'm experiencing a regression between gstreamer 1.16.3 on yocto dunfell with kernel 5.4.149 refering as the "old setup" and gstreamer 1.18.4 on yocto hardknott with kernel 5.10.78 refering as
> the "new
> setup" and am wondering where to dig at it:
> 
> I'm creating a h264 stream on another device with help of v4l2h264enc with:
> 
> gst-launch-1.0 -vvvv v4l2src device=/dev/video/chts_camera_jpeg ! image/jpeg,width=800,height=600,framerate=15/1 ! v4l2jpegdec output-io-mode=dmabuf-import ! v4l2convert output-io-mode=dmabuf-import !
> video/x-raw,co
> lorimetry=bt709 ! v4l2h264enc output-io-mode=dmabuf-import ! 'video/x-h264,level=(string)3.2' ! rtph264pay ! udpsink port=5000 host=192.168.30.114
> 
> This one is accepted with the "old" setup and hardware decoding using:
> 
> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
> videoconvert ! kmssink

Thanks for reporting. I don't really know what is happening. These elements use
the framerate to more accurately estimate their latency, but that is about it.
It is quite uncommon to use such pipeline without do-timestamp on the udpsrc and
an using rtpjitterbuffer. Please consider trying this out, since half backed rtp
pipeline leads to half backed timestamp, which could perhaps be the issue here.
Again, I never obversed this issue myself, so I don't strictly know what is
failing. 

> 
> and also with the old and the new setup with software decoding using:
> 
> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! avdec_h264 ! kmssink
> 
> -----------------------------------------------------------------------------------------
> 
> After patching the kernel with the Patch at "[PATCH v3 1/2] media: coda: Fix reported H264 profile", the stream's caps are accepted, but the stream does not start to run.
> I already found out that the v4lh264enc in combination with coda and imx-hardware does NOT insert vui-parameters in the stream.
> So h264parse will configure the framerate to 0/1.
> 
> The above decoding pipeline will stop at the first still picture. This was also working with the "old setup".
> When simply setting "sync=false" on kmssink the pipeline starts to run, but there is a increasing latency observed because the pictures will flow in with 15 fps but kmssink will only process them with
> about 11 fps, so qeueus will run full.
> 
> I also can avoid this with adding a rate-converter in between, but this solution doesn't look to me as it should be that way.
> 
> Changing the pipeline for using avdec_h264 the problem doesn't exist.
> 
> For my understanding vui-parameters are not mandatory and there may be a bunch of encoders who do not insert them?
> I also have found other people who have similar problems but without a solution at: https://community.nxp.com/t5/i-MX-Processors/vpuenc-framerate-data-missing-vui-parameters-timing-info/m-p/310478.
> 
> When I play content from my pc (x264enc) to that pipelines (with vui-parameters set) all pipelines are working gracefully.
> 
> -------------------------------------------------------------------------------------------
> 
> SUMMARY OF PIPELINES:
> 
> NOT RUNNING (still picture, time stops at 00:00:00)
> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
> v4l2convert ! kmssink
> 
> RUNNING but increasing latency:
> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
> v4l2convert ! kmssink sync = false
> 
> RUNNING but "hacky", guessed framerate of kmssink
> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
> v4l2convert ! queue ! videorate ! "video/x-raw, framerate=10/1" ! kmssink sync = false
> 
> RUNNING but it's a fakesink:
> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
> v4l2convert ! fakesink
> 
> RUNNING but sw-decoding:
> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! avdec_h264 !
> v4l2convert ! kmssink
> 
> 
> My questions:
> Is that vui-information a mandatory requirement for IP-communications standards and may there be a way to insert them with help of other gstreamer-elements "post-encoding".
> 
> If vui-information is not mandatory we would IMHO need to make sure that a "default" gstreamer pipeline without hacky parameters could handle that gracefully.
> Does anybody have an idea what commit could have created that regression so that kmssink stops running now?
> 
> Thanks in advance.
> Best Regards
> Pascal Speck
> 
> 
> 
> 
> 
> 
>