H264 stream without vui-parameters not handled correctly any more

[Nirbheek Chauhan]

Hi Pascal,

Since this is a regression, you may like to file an issue about this
here: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/new

Please include all the information, including the commit that you can
revert to fix it.

Cheers,
Nirbheek

On Fri, Mar 11, 2022 at 6:15 PM Pascal Speck via gstreamer-devel
<gstreamer-devel at lists.freedesktop.org> wrote:
>
> Hi,
> I narrowed it down to the following commit:
>
> https://github.com/GStreamer/gst-plugins-good/commit/613e0a50fe0c3bd87b8e866ab65bc497dc8e9fc8
>
> After that commit this case does not work any more.
> I also printed out the values (before) corresponding line in my case:
>
> info.fps_n : 0, info.fps_d = 1, self->v4l2output->info.fps_n = 1, self->v4l2output->info.fps_d = 65536
>
> So it seems that in this case info is set to a very odd value 1/65536 why ever which leads to the kmssink not working any more.
>
> Currently I don't know how to really fix it without creating a regression to the problem of p.zabel but I'll also have another look on that.
>
> Regards Pascal
>
>
>
> Am 10.03.22 um 20:39 schrieb Nicolas Dufresne via gstreamer-devel:
> > Hi,
> >
> > Le jeudi 10 mars 2022 à 12:06 +0100, gstreamer at iktek.de a écrit :
> >> Hi together,
> >> Again I'm experiencing a regression between gstreamer 1.16.3 on yocto dunfell with kernel 5.4.149 refering as the "old setup" and gstreamer 1.18.4 on yocto hardknott with kernel 5.10.78 refering as
> >> the "new
> >> setup" and am wondering where to dig at it:
> >>
> >> I'm creating a h264 stream on another device with help of v4l2h264enc with:
> >>
> >> gst-launch-1.0 -vvvv v4l2src device=/dev/video/chts_camera_jpeg ! image/jpeg,width=800,height=600,framerate=15/1 ! v4l2jpegdec output-io-mode=dmabuf-import ! v4l2convert output-io-mode=dmabuf-import !
> >> video/x-raw,co
> >> lorimetry=bt709 ! v4l2h264enc output-io-mode=dmabuf-import ! 'video/x-h264,level=(string)3.2' ! rtph264pay ! udpsink port=5000 host=192.168.30.114
> >>
> >> This one is accepted with the "old" setup and hardware decoding using:
> >>
> >> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
> >> videoconvert ! kmssink
> >
> > Thanks for reporting. I don't really know what is happening. These elements use
> > the framerate to more accurately estimate their latency, but that is about it.
> > It is quite uncommon to use such pipeline without do-timestamp on the udpsrc and
> > an using rtpjitterbuffer. Please consider trying this out, since half backed rtp
> > pipeline leads to half backed timestamp, which could perhaps be the issue here.
> > Again, I never obversed this issue myself, so I don't strictly know what is
> > failing.
> >
> >>
> >> and also with the old and the new setup with software decoding using:
> >>
> >> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! avdec_h264 ! kmssink
> >>
> >> -----------------------------------------------------------------------------------------
> >>
> >> After patching the kernel with the Patch at "[PATCH v3 1/2] media: coda: Fix reported H264 profile", the stream's caps are accepted, but the stream does not start to run.
> >> I already found out that the v4lh264enc in combination with coda and imx-hardware does NOT insert vui-parameters in the stream.
> >> So h264parse will configure the framerate to 0/1.
> >>
> >> The above decoding pipeline will stop at the first still picture. This was also working with the "old setup".
> >> When simply setting "sync=false" on kmssink the pipeline starts to run, but there is a increasing latency observed because the pictures will flow in with 15 fps but kmssink will only process them with
> >> about 11 fps, so qeueus will run full.
> >>
> >> I also can avoid this with adding a rate-converter in between, but this solution doesn't look to me as it should be that way.
> >>
> >> Changing the pipeline for using avdec_h264 the problem doesn't exist.
> >>
> >> For my understanding vui-parameters are not mandatory and there may be a bunch of encoders who do not insert them?
> >> I also have found other people who have similar problems but without a solution at: https://community.nxp.com/t5/i-MX-Processors/vpuenc-framerate-data-missing-vui-parameters-timing-info/m-p/310478.
> >>
> >> When I play content from my pc (x264enc) to that pipelines (with vui-parameters set) all pipelines are working gracefully.
> >>
> >> -------------------------------------------------------------------------------------------
> >>
> >> SUMMARY OF PIPELINES:
> >>
> >> NOT RUNNING (still picture, time stops at 00:00:00)
> >> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
> >> v4l2convert ! kmssink
> >>
> >> RUNNING but increasing latency:
> >> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
> >> v4l2convert ! kmssink sync = false
> >>
> >> RUNNING but "hacky", guessed framerate of kmssink
> >> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
> >> v4l2convert ! queue ! videorate ! "video/x-raw, framerate=10/1" ! kmssink sync = false
> >>
> >> RUNNING but it's a fakesink:
> >> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2h264dec !
> >> v4l2convert ! fakesink
> >>
> >> RUNNING but sw-decoding:
> >> gst-launch-1.0 -vvvv udpsrc port=5000 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! avdec_h264 !
> >> v4l2convert ! kmssink
> >>
> >>
> >> My questions:
> >> Is that vui-information a mandatory requirement for IP-communications standards and may there be a way to insert them with help of other gstreamer-elements "post-encoding".
> >>
> >> If vui-information is not mandatory we would IMHO need to make sure that a "default" gstreamer pipeline without hacky parameters could handle that gracefully.
> >> Does anybody have an idea what commit could have created that regression so that kmssink stops running now?
> >>
> >> Thanks in advance.
> >> Best Regards
> >> Pascal Speck
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >