Issues Using GStreamer with v4l2 Elements

[Joshua Quesenberry]

That seems very confusingly labeled, am I missing something in the below high-level description? From an element's perspective, I wouldn't expect to be outputting to my sink pad (upstream) or capturing from my src pad (downstream).

Src element -> v4l2 sink using output-io-mode ... v4l2 src using capture-io-mode -> Sink element

In this example, should the elements dmabuf-import be on the upstream or downstream side? Likewise with the dmabuf? I'm not certain if A) the downstream is creating the dmabuf and the upstream is importing it to dump to or B) the upstream is creating the dmabuf and the downstream is importing it for use... it seems like maybe both work when I try?

Thanks!

JQ

-----Original Message-----
From: Nicolas Dufresne <nicolas at ndufresne.ca> 
Sent: Friday, May 13, 2022 8:47 AM
To: Joshua Quesenberry <engnfrc at gmail.com>
Cc: Discussion of the development of and with GStreamer <gstreamer-devel at lists.freedesktop.org>
Subject: Re: Issues Using GStreamer with v4l2 Elements

Le vendredi 13 mai 2022 à 00:52 -0400, Joshua Quesenberry a écrit :
> Hi Nicolas,
> 
> Sorry for the late reply, I got covered up with other stuff and am 
> just getting a minute to come back to this.
> 
> > Just read your post, looks like you are asking about at least two 
> > distinct issue. In one case, v4l2jpegdec does not work at all for 
> > you. There isn't much information, have you considering prepending 
> > jpegparse in front of it ? That may help ? Jpeg V4L2 decoder is 
> > probably one of the least tested decoder in the v4l2 lot.
> 
> Correct, two separate issues I'm running into.
> 
> v4l2jpegdec works fine with one camera, but not the other. I'm setting 
> them both up very similarly, so it escapes me what could be different 
> to cause this behavior. I just tried jpegparse in the pipeline just 
> before v4l2jpegdec and that didn't have any effect, still the identity 
> just after the decode is never printed. I also tried dropping down 
> from 1280x720 at 60fps to 1024x768 at 30fps to lessen the load a bit and 
> that had no effect either.
> 
> > In the second issue, it works, but its slow. Could it be that you 
> > are loading the CPU ? v4l2h264enc does not automatically do 
> > zero-copy. You can experimentally enable it passing output-io-mode=dmabuf-import.
> 
> I suppose it could be GPU load? When I've looked for ways to measure 
> what the GPU is doing, I've unfortunately not found much of 
> significance. The CPU load is quite low during this testing, <10% for 
> the gstreamer pipeline according to htop. I have a pipeline already 
> with the output-io-modes set to dmabuf-import, but before my original 
> post I switched everything back to auto in hopes things may start 
> working appropriately, however that was not the case. Keep in mind 
> that I have an identity in between the source caps and the capssetter, 
> this is where I see the average framerate drop with v4l2h264enc 
> present.
> 
> I've found that by adding more queues to the pipeline and then setting 
> all queues to have a leaky downstream that the average rate at the 
> source has now climbed back up to what I expected all along, so I'm 
> assuming the issue is a clogged up downstream affecting the upstream 
> as far as the source? After these changes input rate is ~30fps and 
> output to file ~15fps, which is what I'm asking for, and the CPU usage 
> is around ~30%. Playing around with the dmabuf-import option it 
> appears I'm able to get similar rates with a drop in CPU usage to 
> around ~10%, very exciting!
> 
> How do the v4l2 elements work with queues? What do they do when stuck 
> in between a dmabuf-import src pad and dmabuf sink pad? They don't 
> have io-modes to be specified (at least at the user level) and I don't 
> see a v4l2queue or something of the like. Tomorrow I'll do some more 
> experiments to see which queues that I've added are unnecessary.

M2M elements have two queues, so you have output-io-mode (the sink pad) and capture-io-mode (for the src pad).

> 
> So now, camera #1 not getting through v4l2jpegdec is my major 
> roadblock. Anyone have ideas on how to debug this one or know what 
> could cause this behavior?
> 
> FYI, I did dump additional information to the RPi Forums question a 
> few days ago, if something else is needed, just let me know.
> 
> Thanks!
> 
> JQ
> 
> On Mon, May 9, 2022 at 4:15 PM Nicolas Dufresne <nicolas at ndufresne.ca> wrote:
> > 
> > Le lundi 09 mai 2022 à 12:04 -0400, Joshua Quesenberry via 
> > gstreamer-devel a écrit :
> > > Hey Everyone!
> > > 
> > > I’m having some trouble using the v4l2 elements on an RPi. I’ve 
> > > posted a detailed summary on the Raspberry Pi forum. Can anyone 
> > > lend a hand? I’m not sure how many of you are frequent users of 
> > > the RPi, so figured I’d reach out here in parallel in case the RPi forum isn’t the right audience.
> > > https://forums.raspberrypi.com/viewtopic.php?t=334215
> > 
> > Just read your post, looks like you are asking about at least two 
> > distinct issue. In one case, v4l2jpegdec does not work at all for 
> > you. There isn't much information, have you considering prepending 
> > jpegparse in front of it ? That may help ? Jpeg V4L2 decoder is 
> > probably one of the least tested decoder in the v4l2 lot.
> > 
> > In the second issue, it works, but its slow. Could it be that you 
> > are loading the CPU ? v4l2h264enc does not automatically do 
> > zero-copy. You can experimentally enable it passing output-io-mode=dmabuf-import.
> > 
> > Nicolas
> > 
> > > 
> > > Thanks,
> > > 
> > > JQ
> > > 
> > > 
> >