Follow-Up: v4l2h264dec gstreamer 1.18.4 not accepting stream

Wed Mar 9 13:51:00 UTC 2022

Hi Nicolas,

that patch solves the Problem.
Fabio already found that online also so it seems to work the correct way.
As he has more experience getting things upstream he offered me to handle that.
So the issue with the not-accepting-stream is done.

The other issue I already mentioned in that mail is a different one with damaged/missing timing information in the stream.
For short: if one disables the "sync" of the kmssink the stream starts to run.

We'll first try to find out more and come up with another thread if we find something interesting / need further assistance of you/the community.

Up to now: Thanks for your help.

Best Regards
Pascal

Am 08.03.22 um 14:34 schrieb Nicolas Dufresne via gstreamer-devel:
> Le mardi 08 mars 2022 à 08:26 +0100, Pascal Speck via gstreamer-devel a écrit :
>> Hi Nicolas,
>> just for clarification:
>> I'm not working on an RpI, I'm using an i.mx6 customer hardware for getting this running. So the bug may be in the coda driver.
>> Fabio Estevam (who I am also in contact with) already mentioned that the Coda driver did not answer with constrained-basedline ever since it had been introduced although the coda driver supports
> 
> Ok, threads started piling up confusingly. I've report that specific issue to
> the maintainer (Philipp Zabel) few years ago. Meanwhile, I wrote a quick patch
> before 1.18, but as I don't know what I'm doing and didn't have the time to
> check if it was correct yet, it wasn't sent. Here's the patches in question, I
> have no idea on what this is based on nor if its a proper fix.
> 
>> constrained baseline. So yes: this is a bug which still has not been hit until gst 1.18.0
>> I already patched the driver on my local host to answer constrained-baseline also.
>> This leads to a valid pipeline setup but unfortunately the pipeline does not "RUN".
>> It shows a (successfully decoded) still picture of the camera and stops at 0:00:00 / 99:99:99 in the PLAYING state.
>> So I also have to start to dig into this issue now.
>>
>> Do you have a starting point for investigating this issue? (avdec_h264 produces a "really running" pipeline so I think the issue is also somewhere in the v4l2h264<->coda area).
>>
>> As soon as I have a working setup I'll propose a Patch to the imx/coda folks to add constrained-baseline to the v4l2 driver.
>>
>> Best Regards Pascal
>>
>> Am 07.03.22 um 19:39 schrieb Nicolas Dufresne via gstreamer-devel:
>>> Le lundi 07 mars 2022 à 15:53 +0100, Pascal Speck via gstreamer-devel a écrit :
>>>> Am 07.03.22 um 15:41 schrieb Nicolas Dufresne via gstreamer-devel:
>>>>> Le lundi 07 mars 2022 à 12:29 +0100, Pascal Speck via gstreamer-devel a écrit :
>>>>>> Hi together...
>>>>>>
>>>>>> After a bit more of digging I found out that it seems to be the profile cap which is not accepted here:
>>>>>
>>>>> Ok.
>>>>>
>>>>>>
>>>>>> I see the "gst_caps_set_subset" beeing called with the following:
>>>>>>
>>>>>> caps video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)800, height=(int)600, framerate=(fraction)0/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8,
>>>>>> bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)3.1
>>>>>>
>>>>>> template_caps: video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string){ 1, 1b, 1.1, 1.2, 1.3, 2, 2.1, 2.2, 3, 3.1, 3.2, 4, 4.1 }, profile=(string){ baseline, main, high }
>>>>>
>>>>> This is a driver bug, mainline encoder must support constrained baseline. The
>>>>> driver must be fixed because it is stateful and it will not mark the constraint
>>>>> in the bitstream otherwise.
>>>>
>>>> So if I get you right the v4l2h264dec needs to report something like  { constrained-baseline, basline, constrained-main, main, constrained-high } instead of { baseline, main, high } in his caps, so
>>>> the above stream-caps would get accepted.
>>>> I'm wondering where the regression from the old setup happened.. I'll try to show the caps of the old setup which are reported by the driver and find the point where they're reported wrong.
>>>
>>> Older GStreamer version was completely ignoring V4L2 profiles. We needed to
>>> exposed the profile in the template so that auto-plugger can fallback if the
>>> selected HW (encoder or decoder) is not capable of the the profile requested in
>>> the caps. This feature was enabled in 1.18.0 (8 September 2020), so in theory,
>>> RPi folks knew about this and just ignored it. Or they just don't test much the
>>> multimedia side.
>>>
>>> It has been the same for many years, each time we start using a feature in V4L2,
>>> we hit drivers with partial or incorrect implementation. We support the "not
>>> implemented" cases (unless we have a bug), but not the partially implemented
>>> one. As we speak, we started working on Kernel CI, so that we can test newer
>>> Kernel against current GStreamer and vis-versa, but RPi folks won't benefit of
>>> this effort until they actually land their decoder driver(s) upstream. So they
>>> highly rely on users for the testing and debugging.
>>>
>>
>>
>>>>
>>>>>
>>>>>>
>>>>>> As this function does only compare for "equal" features, I would assume that it breaks here.
>>>>>>
>>>>>> I'm currently wondering who does the mistake here:
>>>>>>
>>>>>> Should the h264parse element also add the profile="baseline" feature or should the decoder also answer "constrained-baseline" in it's caps to fullfil this issue.?!
>>>>>>
>>>>>> Some help of the gst-experts what should be the expected behaviour is appreciated here, so I would know at which module I should search for changes.
>>>>>>
>>>>>> Best Regards
>>>>>>
>>>>>> Pascal
>>>>>>
>>>>>> Am 07.03.22 um 08:18 schrieb Pascal Speck via gstreamer-devel:
>>>>>>>
>>>>>>> Currently I'm wondering about 3 things:
>>>>>>>
>>>>>>> 1: Is it really the colorimetry that is not accepted?
>>>>>>> I don't see the "ERROR: from element /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0:
>>>>>>> Device '/dev/xxx' does not support xxx colorimetry" message here. I was assuming into the direction of "baseline" vs "constrained-baseline" because v4l2h264enc is only returning baseline in accepted caps.
>>>>>>>
>>>>>>> 2: Can one give me an idea how to gain more logging information about which capability is causing the "non-acceptance"? Would it be a deal to generally having this information in the gst-error-log?
>>>>>>>
>>>>>>> I'll try to dig in further meanwhile but as I'm not a gstreamer-expert I first have to find the file where it's worth starting to dig into the code... :-)
>>>>>>>
>>>>>>> 3: Is there a simple solution to just override the caps in the pipeline only "downstream" to pretend to have a different colorimetry or profile.. If so I would maybe findout more which of the caps is
>>>>>>> not accepted...
>>>>>>>
>>>>>>> Regards Pascal
>>>>>>>
>>>>>>> Am 04.03.22 um 14:52 schrieb Nicolas Dufresne via gstreamer-devel:
>>>>>>> Am 04.03.22 um 14:52 schrieb Nicolas Dufresne via gstreamer-devel:
>>>>>>>
>>>>>>>> Le vendredi 04 mars 2022 à 11:21 +0100, Pascal Speck via gstreamer-devel a
>>>>>>>> écrit :
>>>>>>>>>
>>>>>>>>> 0:00:03.690824695  1500  0xffa000 INFO                    v4l2 gstv4l2object.c:4469:gst_v4l2_object_probe_caps:<v4l2h264dec0:sink> probed caps: video/x-h264, stream-format=(string)byte-stream,
>>>>>>>>> alignment=(string)au, level=(string){ 1, 1
>>>>>>>>> b, 1.1, 1.2, 1.3, 2, 2.1, 2.2, 3, 3.1, 3.2, 4, 4.1 }, profile=(string){ baseline, main, high }, width=(int)[ 48, 1920 ], height=(int)[ 16, 1088 ], colorimetry=(string){ bt709, bt601, smpte240m,
>>>>>>>>> 2:4:5:2, 2:4:5:3, 1:4:7:1, 2:4:7:1, 2:4:12
>>>>>>>>> :8, bt2020, 2:0:0:0 }, parsed=(boolean)true
>>>>>>>>>
>>>>>>>>> I can see that with the old setup the caps seem very similar but are accepted:
>>>>>>>>>
>>>>>>>>> /GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)800, height=(int)600, framerate=(fraction)0/1,
>>>>>>>>> interlace-mode=(string)progressive, chrom
>>>>>>>>> a-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true, profile=(string)constrained-baseline, level=(string)3.1
>>>>>>>>>
>>>>>>>>> Does someone have an Idea where to dig further?
>>>>>>>>
>>>>>>>> Looks quite similar to:
>>>>>>>>
>>>>>>>> https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1056
>>>>>>>>
>>>>>>>> We *just* need someone with free time to fix it.
>>>>>>>>
>>>>>>>> Nicolas
>>>>>
>>>
> 