H264 stream without vui-parameters not handled correctly any more

Fri Mar 11 11:36:50 UTC 2022

Hi,
I narrowed it down to the following commit:

https://github.com/GStreamer/gst-plugins-good/commit/613e0a50fe0c3bd87b8e866ab65bc497dc8e9fc8

After that commit this case does not work any more.
I also printed out the values (before) corresponding line in my case:

info.fps_n : 0, info.fps_d = 1, self->v4l2output->info.fps_n = 1, self->v4l2output->info.fps_d = 65536

So it seems that in this case info is set to a very odd value 1/65536 why ever which leads to the kmssink not working any more.

Currently I don't know how to really fix it without creating a regression to the problem of p.zabel but I'll also have another look on that.

Regards Pascal

Am 10.03.22 um 20:39 schrieb Nicolas Dufresne via gstreamer-devel:
> Hi,
> 
> Le jeudi 10 mars 2022 à 12:06 +0100, gstreamer at iktek.de a écrit :
>> Hi together,
>> Again I'm experiencing a regression between gstreamer 1.16.3 on yocto dunfell with kernel 5.4.149 refering as the "old setup" and gstreamer 1.18.4 on yocto hardknott with kernel 5.10.78 refering as
>> the "new
>> setup" and am wondering where to dig at it:
>>
>> I'm creating a h264 stream on another device with help of v4l2h264enc with:
>>
>> gst-launch-1.0 -vvvv v4l2src device=/dev/video/chts_camera_jpeg ! image/jpeg,width=800,height=600,framerate=15/1 ! v4l2jpegdec output-io-mode=dmabuf-import ! v4l2convert output-io-mode=dmabuf-import !
>> video/x-raw,co
>> lorimetry=bt709 ! v4l2h264enc output-io-mode=dmabuf-import ! 'video/x-h264,level=(string)3.2' ! rtph264pay ! udpsink port=5000 host=192.168.30.114
>>
>> This one is accepted with the "old" setup and hardware decoding using:
>>
>> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
>> videoconvert ! kmssink
> 
> Thanks for reporting. I don't really know what is happening. These elements use
> the framerate to more accurately estimate their latency, but that is about it.
> It is quite uncommon to use such pipeline without do-timestamp on the udpsrc and
> an using rtpjitterbuffer. Please consider trying this out, since half backed rtp
> pipeline leads to half backed timestamp, which could perhaps be the issue here.
> Again, I never obversed this issue myself, so I don't strictly know what is
> failing. 
> 
>>
>> and also with the old and the new setup with software decoding using:
>>
>> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! avdec_h264 ! kmssink
>>
>> -----------------------------------------------------------------------------------------
>>
>> After patching the kernel with the Patch at "[PATCH v3 1/2] media: coda: Fix reported H264 profile", the stream's caps are accepted, but the stream does not start to run.
>> I already found out that the v4lh264enc in combination with coda and imx-hardware does NOT insert vui-parameters in the stream.
>> So h264parse will configure the framerate to 0/1.
>>
>> The above decoding pipeline will stop at the first still picture. This was also working with the "old setup".
>> When simply setting "sync=false" on kmssink the pipeline starts to run, but there is a increasing latency observed because the pictures will flow in with 15 fps but kmssink will only process them with
>> about 11 fps, so qeueus will run full.
>>
>> I also can avoid this with adding a rate-converter in between, but this solution doesn't look to me as it should be that way.
>>
>> Changing the pipeline for using avdec_h264 the problem doesn't exist.
>>
>> For my understanding vui-parameters are not mandatory and there may be a bunch of encoders who do not insert them?
>> I also have found other people who have similar problems but without a solution at: https://community.nxp.com/t5/i-MX-Processors/vpuenc-framerate-data-missing-vui-parameters-timing-info/m-p/310478.
>>
>> When I play content from my pc (x264enc) to that pipelines (with vui-parameters set) all pipelines are working gracefully.
>>
>> -------------------------------------------------------------------------------------------
>>
>> SUMMARY OF PIPELINES:
>>
>> NOT RUNNING (still picture, time stops at 00:00:00)
>> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
>> v4l2convert ! kmssink
>>
>> RUNNING but increasing latency:
>> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
>> v4l2convert ! kmssink sync = false
>>
>> RUNNING but "hacky", guessed framerate of kmssink
>> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
>> v4l2convert ! queue ! videorate ! "video/x-raw, framerate=10/1" ! kmssink sync = false
>>
>> RUNNING but it's a fakesink:
>> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
>> v4l2convert ! fakesink
>>
>> RUNNING but sw-decoding:
>> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! avdec_h264 !
>> v4l2convert ! kmssink
>>
>>
>> My questions:
>> Is that vui-information a mandatory requirement for IP-communications standards and may there be a way to insert them with help of other gstreamer-elements "post-encoding".
>>
>> If vui-information is not mandatory we would IMHO need to make sure that a "default" gstreamer pipeline without hacky parameters could handle that gracefully.
>> Does anybody have an idea what commit could have created that regression so that kmssink stops running now?
>>
>> Thanks in advance.
>> Best Regards
>> Pascal Speck
>>
>>
>>
>>
>>
>>
>>
> 