Issues Using GStreamer with v4l2 Elements

Mon Aug 22 18:17:42 UTC 2022

Hi Josh,

Le lundi 22 août 2022 à 13:23 -0400, Joshua Quesenberry a écrit :
> Thanks Nicolas. I've forwarded that information over to my RPi forums post. I don't suppose you have a ticket # available that I can reference?

Can't recall exactly, since a lot of the information got shared to me through
chats (LibreELEC Slack and GSTreamer IRC) while trying to get this bigger issue
sorted:

https://github.com/raspberrypi/firmware/issues/1673

I know a lot of firmware fixes were being shipped in parallel to this series of
backport.

> 
> Thanks,
> 
> Josh Q
> 
> -----Original Message-----
> From: Nicolas Dufresne <nicolas at ndufresne.ca> 
> Sent: Monday, August 22, 2022 1:09 PM
> To: Joshua Quesenberry <engnfrc at gmail.com>
> Cc: 'Discussion of the development of and with GStreamer' <gstreamer-devel at lists.freedesktop.org>
> Subject: Re: Issues Using GStreamer with v4l2 Elements
> 
> Hi Joshua,
> 
> Le lundi 22 août 2022 à 10:58 -0400, Joshua Quesenberry a écrit :
> > Nicolas,
> > 
> > Please find attached zip files for a jpeg capture and a raw capture.
> 
> From a quick look, my impression is that the driver does delay the end of the draining flow (CMD_STOP or bytesused = 0) to after the last decoded buffer has been dequeued. From your trace, you can observe this by a chain from the identity_pre and not for the identify_post.
> 
> I believe this is an issue that was previously reported to RPi maintainers and should in theory be known or addressed.
> 
> regards,
> Nicolas
> 
> 
> > 
> > Thanks!
> > 
> > Josh Q
> > 
> > -----Original Message-----
> > From: Nicolas Dufresne <nicolas at ndufresne.ca>
> > Sent: Monday, August 22, 2022 9:34 AM
> > To: Joshua Quesenberry <engnfrc at gmail.com>
> > Cc: 'Discussion of the development of and with GStreamer' 
> > <gstreamer-devel at lists.freedesktop.org>
> > Subject: Re: Issues Using GStreamer with v4l2 Elements
> > 
> > Le lundi 22 août 2022 à 09:13 -0400, Joshua Quesenberry a écrit :
> > > Is anyone able to assist me with overcoming this issue? I’m 
> > > currently unable to use a particular make/model of camera 
> > > (ELP-USBFHD01M w/
> > > OV2710) with the v4l2jpegdec element. I’ve had other cameras work 
> > > with it. I don’t notice any errors when upping the GST_DEBUG options.
> > > Traffic is at the input pad of the element, but nothing coming from 
> > > the output pad. I have some logs posted on the RPi forum issue I posted:
> > > https://forums.raspberrypi.com/viewtopic.php?p=2031632#p2031632
> > 
> > Perhaps you should still share the pipeline (in gst-launch syntax) that you are trying to use ? As you don't see any errors, perhaps you should store to disk a snippet of JPEG and share that too ? As it is now, I don't see much I can add to help you.
> > 
> > >  
> > > Thanks,
> > >  
> > > Josh Q
> > >  
> > > From: Joshua Quesenberry <engnfrc at gmail.com>
> > > Sent: Sunday, August 14, 2022 9:00 AM
> > > To: Nicolas Dufresne <nicolas at ndufresne.ca>
> > > Cc: Discussion of the development of and with GStreamer 
> > > <gstreamer-devel at lists.freedesktop.org>; Joshua Quesenberry 
> > > <EngnFrc at gmail.com>
> > > Subject: Re: Issues Using GStreamer with v4l2 Elements
> > >  
> > > I’m coming back to this finally. I’ve yet to find a solution to 
> > > v4l2jpegdec not working with this specific make/model camera. It works fine with jpegdec.
> > > With v4l2jpegdec there’s no downstream data, observable using 
> > > identity element, but I do see data upstream using identity. I’m not 
> > > seeing any errors in dmesg or in the gstreamer output if I up the debug levels.
> > > What else can be done to try and figure out why things aren’t working?
> > >  
> > > Thanks,
> > >  
> > > Josh Q
> > >  
> > > On Fri, May 13, 2022 at 4:46 PM Nicolas Dufresne <nicolas at ndufresne.ca> wrote:
> > > > Le vendredi 13 mai 2022 à 14:28 -0400, Joshua Quesenberry a écrit :
> > > > > That seems very confusingly labeled, am I missing something in 
> > > > > the below high-level description? From an element's perspective, 
> > > > > I wouldn't expect to be outputting to my sink pad (upstream) or 
> > > > > capturing from my src pad (downstream).
> > > > > 
> > > > > Src element -> v4l2 sink using output-io-mode ... v4l2 src using
> > > > > capture- io-mode -> Sink element
> > > > > 
> > > > > In this example, should the elements dmabuf-import be on the 
> > > > > upstream or downstream side? Likewise with the dmabuf? I'm not 
> > > > > certain if A) the downstream is creating the dmabuf and the 
> > > > > upstream is importing it to dump to or B) the upstream is 
> > > > > creating the dmabuf and the downstream is importing it for 
> > > > > use... it seems like maybe both work when I try?
> > > > 
> > > > I don't see a pipeline description I could comment on. By default, 
> > > > all v4l2 srcpad are dmabuf exporters. But the sink pad are not 
> > > > automatically importer, as this code is experimental and need some 
> > > > polishing and testing.
> > > > 
> > > > > 
> > > > > Thanks!
> > > > > 
> > > > > JQ
> > > > > 
> > > > > -----Original Message-----
> > > > > From: Nicolas Dufresne <nicolas at ndufresne.ca>
> > > > > Sent: Friday, May 13, 2022 8:47 AM
> > > > > To: Joshua Quesenberry <engnfrc at gmail.com>
> > > > > Cc: Discussion of the development of and with GStreamer 
> > > > > <gstreamer-devel at lists.freedesktop.org>
> > > > > Subject: Re: Issues Using GStreamer with v4l2 Elements
> > > > > 
> > > > > Le vendredi 13 mai 2022 à 00:52 -0400, Joshua Quesenberry a écrit :
> > > > > > Hi Nicolas,
> > > > > > 
> > > > > > Sorry for the late reply, I got covered up with other stuff 
> > > > > > and am just getting a minute to come back to this.
> > > > > > 
> > > > > > > Just read your post, looks like you are asking about at 
> > > > > > > least two distinct issue. In one case, v4l2jpegdec does not 
> > > > > > > work at all for you. There isn't much information, have you 
> > > > > > > considering prepending jpegparse in front of it ? That may 
> > > > > > > help ? Jpeg V4L2 decoder is probably one of the least tested decoder in the v4l2 lot.
> > > > > > 
> > > > > > Correct, two separate issues I'm running into.
> > > > > > 
> > > > > > v4l2jpegdec works fine with one camera, but not the other. I'm 
> > > > > > setting them both up very similarly, so it escapes me what 
> > > > > > could be different to cause this behavior. I just tried 
> > > > > > jpegparse in the pipeline just before v4l2jpegdec and that 
> > > > > > didn't have any effect, still the identity just after the 
> > > > > > decode is never printed. I also tried dropping down from 
> > > > > > 1280x720 at 60fps to 1024x768 at 30fps to lessen the load a bit and that had no effect either.
> > > > > > 
> > > > > > > In the second issue, it works, but its slow. Could it be 
> > > > > > > that you are loading the CPU ? v4l2h264enc does not 
> > > > > > > automatically do zero-copy. You can experimentally enable it 
> > > > > > > passing
> > > > > > > output-io- mode=dmabuf-import.
> > > > > > 
> > > > > > I suppose it could be GPU load? When I've looked for ways to 
> > > > > > measure what the GPU is doing, I've unfortunately not found 
> > > > > > much of significance. The CPU load is quite low during this 
> > > > > > testing, <10% for the gstreamer pipeline according to htop. I 
> > > > > > have a pipeline already with the output-io-modes set to 
> > > > > > dmabuf-import, but before my original post I switched 
> > > > > > everything back to auto in hopes things may start working 
> > > > > > appropriately, however that was not the case. Keep in mind 
> > > > > > that I have an identity in between the source caps and the 
> > > > > > capssetter, this is where I see the average framerate drop with v4l2h264enc present.
> > > > > > 
> > > > > > I've found that by adding more queues to the pipeline and then 
> > > > > > setting all queues to have a leaky downstream that the average 
> > > > > > rate at the source has now climbed back up to what I expected 
> > > > > > all along, so I'm assuming the issue is a clogged up 
> > > > > > downstream affecting the upstream as far as the source? After 
> > > > > > these changes input rate is ~30fps and output to file ~15fps, 
> > > > > > which is what I'm asking for, and the CPU usage is around 
> > > > > > ~30%. Playing around with the dmabuf-import option it appears 
> > > > > > I'm able to get similar rates with a drop in CPU usage to around ~10%, very exciting!
> > > > > > 
> > > > > > How do the v4l2 elements work with queues? What do they do 
> > > > > > when stuck in between a dmabuf-import src pad and dmabuf sink pad?
> > > > > > They don't have io-modes to be specified (at least at the user
> > > > > > level) and I don't see a v4l2queue or something of the like. 
> > > > > > Tomorrow I'll do some more experiments to see which queues that I've added are unnecessary.
> > > > > 
> > > > > M2M elements have two queues, so you have output-io-mode (the 
> > > > > sink
> > > > > pad) and capture-io-mode (for the src pad).
> > > > > 
> > > > > > 
> > > > > > So now, camera #1 not getting through v4l2jpegdec is my major 
> > > > > > roadblock. Anyone have ideas on how to debug this one or know 
> > > > > > what could cause this behavior?
> > > > > > 
> > > > > > FYI, I did dump additional information to the RPi Forums 
> > > > > > question a few days ago, if something else is needed, just let me know.
> > > > > > 
> > > > > > Thanks!
> > > > > > 
> > > > > > JQ
> > > > > > 
> > > > > > On Mon, May 9, 2022 at 4:15 PM Nicolas Dufresne 
> > > > > > <nicolas at ndufresne.ca>
> > > > > > wrote:
> > > > > > > 
> > > > > > > Le lundi 09 mai 2022 à 12:04 -0400, Joshua Quesenberry via 
> > > > > > > gstreamer-devel a écrit :
> > > > > > > > Hey Everyone!
> > > > > > > > 
> > > > > > > > I’m having some trouble using the v4l2 elements on an RPi. 
> > > > > > > > I’ve posted a detailed summary on the Raspberry Pi forum. 
> > > > > > > > Can anyone lend a hand? I’m not sure how many of you are 
> > > > > > > > frequent users of the RPi, so figured I’d reach out here 
> > > > > > > > in parallel in case the RPi forum isn’t the right audience.
> > > > > > > > https://forums.raspberrypi.com/viewtopic.php?t=334215
> > > > > > > 
> > > > > > > Just read your post, looks like you are asking about at 
> > > > > > > least two distinct issue. In one case, v4l2jpegdec does not 
> > > > > > > work at all for you. There isn't much information, have you 
> > > > > > > considering prepending jpegparse in front of it ? That may 
> > > > > > > help ? Jpeg V4L2 decoder is probably one of the least tested decoder in the v4l2 lot.
> > > > > > > 
> > > > > > > In the second issue, it works, but its slow. Could it be 
> > > > > > > that you are loading the CPU ? v4l2h264enc does not 
> > > > > > > automatically do zero-copy. You can experimentally enable it 
> > > > > > > passing
> > > > > > > output-io- mode=dmabuf-import.
> > > > > > > 
> > > > > > > Nicolas
> > > > > > > 
> > > > > > > > 
> > > > > > > > Thanks,
> > > > > > > > 
> > > > > > > > JQ
> > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > 
> > > > > 
> > 
> 
> 