FollowUp: H264 stream without vui-parameters not handled correctly any more

Mon Mar 14 08:31:19 UTC 2022

Hi at List: again a little follow up with more information:

The wrong values orginate from here:

-------------------------------------------

gstv4l2object.c @ line 3915:

....
 streamparm.parm.output.timeperframe.numerator = fps_d;
    streamparm.parm.output.timeperframe.denominator = fps_n;

    if (v4l2object->ioctl (fd, VIDIOC_S_PARM, &streamparm) < 0)
      goto set_parm_failed;

    if (streamparm.parm.output.timeperframe.numerator > 0 &&
        streamparm.parm.output.timeperframe.denominator > 0) {
      /* get new values */
      fps_d = streamparm.parm.output.timeperframe.numerator;
      fps_n = streamparm.parm.output.timeperframe.denominator;
....

Tries to set 0/1 to the encoder and after that takes the fps_d and fps_n values from the IOCL answer.

--------------------------------------

drivers/media/platform/coda/coda-common.c @ line 1369ff:

Here a bounding takes place:
....
	/* Lower bound is 1/65535 */
	if (s.numerator == 0 || s.denominator / s.numerator > 65535) {
		timeperframe->numerator = 1;
		timeperframe->denominator = 65535;
		return;
	}

	/* Upper bound is 65536/1 */
	if (s.denominator == 0 || s.numerator / s.denominator > 65536) {
		timeperframe->numerator = 65536;
		timeperframe->denominator = 1;
		return;
	}
....
so the framerate 0/1 is "bounded" to 1/65536. So for coda 0/1 does not seem to be something "valid" and the S_PARAM iocl is then returning 1/65535 which is then not accepted by the rest of gstreamer
because it's a valid framerate and not an "unknown" one...

----------------------------------------

I really don't know which is the correct way to fix this:

1. fix coda to allow a 0/1 as a "valid" framerate.
2. fix gstv4l2object.c to not take the bounded framerate.
3. something else...

Regards Pascal

Am 11.03.22 um 12:36 schrieb Pascal Speck via gstreamer-devel:
> Hi,
> I narrowed it down to the following commit:
> 
> https://github.com/GStreamer/gst-plugins-good/commit/613e0a50fe0c3bd87b8e866ab65bc497dc8e9fc8
> 
> After that commit this case does not work any more.
> I also printed out the values (before) corresponding line in my case:
> 
> info.fps_n : 0, info.fps_d = 1, self->v4l2output->info.fps_n = 1, self->v4l2output->info.fps_d = 65536
> 
> So it seems that in this case info is set to a very odd value 1/65536 why ever which leads to the kmssink not working any more.
> 
> Currently I don't know how to really fix it without creating a regression to the problem of p.zabel but I'll also have another look on that.
> 
> Regards Pascal
> 
> 
> 
> Am 10.03.22 um 20:39 schrieb Nicolas Dufresne via gstreamer-devel:
>> Hi,
>>
>> Le jeudi 10 mars 2022 à 12:06 +0100, gstreamer at iktek.de a écrit :
>>> Hi together,
>>> Again I'm experiencing a regression between gstreamer 1.16.3 on yocto dunfell with kernel 5.4.149 refering as the "old setup" and gstreamer 1.18.4 on yocto hardknott with kernel 5.10.78 refering as
>>> the "new
>>> setup" and am wondering where to dig at it:
>>>
>>> I'm creating a h264 stream on another device with help of v4l2h264enc with:Just for your interesst:
>>>
>>> gst-launch-1.0 -vvvv v4l2src device=/dev/video/chts_camera_jpeg ! image/jpeg,width=800,height=600,framerate=15/1 ! v4l2jpegdec output-io-mode=dmabuf-import ! v4l2convert output-io-mode=dmabuf-import !
>>> video/x-raw,co
>>> lorimetry=bt709 ! v4l2h264enc output-io-mode=dmabuf-import ! 'video/x-h264,level=(string)3.2' ! rtph264pay ! udpsink port=5000 host=192.168.30.114
>>>
>>> This one is accepted with the "old" setup and hardware decoding using:
>>>
>>> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
>>> videoconvert ! kmssink
>>
>> Thanks for reporting. I don't really know what is happening. These elements use
>> the framerate to more accurately estimate their latency, but that is about it.
>> It is quite uncommon to use such pipeline without do-timestamp on the udpsrc and
>> an using rtpjitterbuffer. Please consider trying this out, since half backed rtp
>> pipeline leads to half backed timestamp, which could perhaps be the issue here.
>> Again, I never obversed this issue myself, so I don't strictly know what is
>> failing. 
>>
>>>
>>> and also with the old and the new setup with software decoding using:
>>>
>>> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! avdec_h264 ! kmssink
>>>
>>> -----------------------------------------------------------------------------------------
>>>
>>> After patching the kernel with the Patch at "[PATCH v3 1/2] media: coda: Fix reported H264 profile", the stream's caps are accepted, but the stream does not start to run.
>>> I already found out that the v4lh264enc in combination with coda and imx-hardware does NOT insert vui-parameters in the stream.
>>> So h264parse will configure the framerate to 0/1.
>>>
>>> The above decoding pipeline will stop at the first still picture. This was also working with the "old setup".
>>> When simply setting "sync=false" on kmssink the pipeline starts to run, but there is a increasing latency observed because the pictures will flow in with 15 fps but kmssink will only process them with
>>> about 11 fps, so qeueus will run full.
>>>
>>> I also can avoid this with adding a rate-converter in between, but this solution doesn't look to me as it should be that way.
>>>
>>> Changing the pipeline for using avdec_h264 the problem doesn't exist.
>>>
>>> For my understanding vui-parameters are not mandatory and there may be a bunch of encoders who do not insert them?
>>> I also have found other people who have similar problems but without a solution at: https://community.nxp.com/t5/i-MX-Processors/vpuenc-framerate-data-missing-vui-parameters-timing-info/m-p/310478.
>>>
>>> When I play content from my pc (x264enc) to that pipelines (with vui-parameters set) all pipelines are working gracefully.
>>>
>>> -------------------------------------------------------------------------------------------
>>>
>>> SUMMARY OF PIPELINES:
>>>
>>> NOT RUNNING (still picture, time stops at 00:00:00)
>>> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
>>> v4l2convert ! kmssink
>>>
>>> RUNNING but increasing latency:
>>> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
>>> v4l2convert ! kmssink sync = false
>>>
>>> RUNNING but "hacky", guessed framerate of kmssink
>>> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
>>> v4l2convert ! queue ! videorate ! "video/x-raw, framerate=10/1" ! kmssink sync = false
>>>
>>> RUNNING but it's a fakesink:
>>> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
>>> v4l2convert ! fakesink
>>>
>>> RUNNING but sw-decoding:
>>> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! avdec_h264 !
>>> v4l2convert ! kmssink
>>>
>>>
>>> My questions:
>>> Is that vui-information a mandatory requirement for IP-communications standards and may there be a way to insert them with help of other gstreamer-elements "post-encoding".
>>>
>>> If vui-information is not mandatory we would IMHO need to make sure that a "default" gstreamer pipeline without hacky parameters could handle that gracefully.
>>> Does anybody have an idea what commit could have created that regression so that kmssink stops running now?
>>>
>>> Thanks in advance.
>>> Best Regards
>>> Pascal Speck
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>