encoding pipeline from v4l2 usb camera with v4l2video8convert and v4l2h264enc

Mon Jan 20 19:32:01 UTC 2020

Le lundi 20 janvier 2020 à 20:13 +0200, Constantine Elster a écrit :
> Hi Nicolas,
> 
> Thank you very much!! 
> 
> I removed output-io-mode=dmabuf-import from v4l2video8convert and v4l2h264enc
> elements and got the pipeline working!
> This pipeline worked for me: gst-launch-1.0 -vvv v4l2src device="/dev/video9"
> num-buffers=1000 ! "video/x-raw, format=(string)UYVY, width=(int)640,
> height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)45/1,
> colorimetry=bt709" ! v4l2video8convert ! v4l2h264enc ! h264parse ! mp4mux !
> filesink location=aha.mp4
> 
> I get about 15% CPU usage with the above pipeline, though I get few dropped
> frames in the output file.

That might go away with a 'queue' between v4l2src and v4l2video8convert.

> 
> So, the next step I want to perform few optimizations.
> 1) I set the io-mode=dmabuf-import on the v4l2src element though I get this
> error: 
> ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Failed to
> allocate required memory.
> Additional debug info:
> gstv4l2src.c(658): gst_v4l2src_decide_allocation ():
> /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
> Buffer pool activation failed
> 
> So, I presume 0memcopy is not possible?

I guess not indeed. Would be nice to learn more (GST_DEBUG="v4l2*:7"). Though I
know UVC driver is not flexible at all. Also, buffer importation code isn't safe
in older release of GStreamer. On the good side, the copy is made with a
virtually allocated buffer as source, so it fairly fast.

> 
> 2) I would like to drop frames as close as possible to the v4l2 source to
> achieve 10 fps at the input of the encoder. Though my v4l2 src accepts only 45
> and 60 fps configuration. Is there an elegant way to reduce the frame rate
> this way?

I'd use "videorate drop-only=0 ! <caps filter with framerate=10/1>". 

> 
> 3) Any other recommendations I can pursue to optimize the pipeline?

You can try to enable 0 copy between v4l2video8convert and v4l2h264enc. Two
options:

   v4l2video8convert capture-io-mode=dmabuf-import ! v4l2h264enc
   v4l2video8convert ! v4l2h264enc output-io-mode=dmabuf-import

> 
> Thank you very very much,
>   -- Constantine.
> 
> 
> On Sun, Jan 19, 2020 at 7:57 PM Nicolas Dufresne <nicolas at ndufresne.ca> wrote:
> > Le dimanche 19 janvier 2020 à 15:12 +0200, Constantine Elster a écrit :
> > > Thank you very much Milian! Looks better with h264parse. Still getting an
> > error, though now different one.
> > > 
> > > I added h264parse to the pipeline:
> > > gst-launch-1.0 -vvv v4l2src device="/dev/video9" num-buffers=200 !
> > "video/x-raw, format=(string)UYVY, width=(int)640, height=(int)480, pixel-
> > aspect-ratio=(fraction)1/1, framerate=(fraction)60/1, colorimetry=bt709" !
> > v4l2video8convert output-io-mode=dmabuf-import ! v4l2h264enc output-io-
> > mode=dmabuf-import ! h264parse ! mp4mux ! filesink location=aha.mp4
> > > 
> > > Output:
> > > Setting pipeline to PAUSED ...
> > > Pipeline is live and does not need PREROLL ...
> > > Setting pipeline to PLAYING ...
> > > New clock: GstSystemClock
> > > /GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, 
> > format=(string)UYVY, width=(int)640, height=(int)480, pixel-aspect-
> > ratio=(fraction)1/1, framerate=(fraction)60/1, colorimetry=(string)bt709,
> > interlace-mode=(string)progressive
> > > /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps =
> > video/x-raw, format=(string)UYVY, width=(int)640, height=(int)480, pixel-
> > aspect-ratio=(fraction)1/1, framerate=(fraction)60/1,
> > colorimetry=(string)bt709, interlace-mode=(string)progressive
> > > /GstPipeline:pipeline0/v4l2video8convert:v4l2video8convert0.GstPad:src:
> > caps = video/x-raw, framerate=(fraction)60/1, interlace-
> > mode=(string)progressive, format=(string)I420, width=(int)640,
> > height=(int)480, colorimetry=(string)bt709
> > > /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0.GstPad:src: caps =
> > video/x-h264, stream-format=(string)byte-stream, alignment=(string)au,
> > profile=(string)baseline, level=(string)4, width=(int)640, height=(int)480,
> > pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1, interlace-
> > mode=(string)progressive, colorimetry=(string)bt709
> > > /GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:sink: caps =
> > video/x-h264, stream-format=(string)byte-stream, alignment=(string)au,
> > profile=(string)baseline, level=(string)4, width=(int)640, height=(int)480,
> > pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1, interlace-
> > mode=(string)progressive, colorimetry=(string)bt709
> > > Redistribute latency...
> > > /GstPipeline:pipeline0/v4l2h264enc:v4l2h264enc0.GstPad:sink: caps =
> > video/x-raw, framerate=(fraction)60/1, interlace-mode=(string)progressive,
> > format=(string)I420, width=(int)640, height=(int)480,
> > colorimetry=(string)bt709
> > > /GstPipeline:pipeline0/v4l2video8convert:v4l2video8convert0.GstPad:sink:
> > caps = video/x-raw, format=(string)UYVY, width=(int)640, height=(int)480,
> > pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1,
> > colorimetry=(string)bt709, interlace-mode=(string)progressive
> > > /GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps =
> > video/x-raw, format=(string)UYVY, width=(int)640, height=(int)480, pixel-
> > aspect-ratio=(fraction)1/1, framerate=(fraction)60/1,
> > colorimetry=(string)bt709, interlace-mode=(string)progressive
> > > ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal
> > data stream error.
> > > Additional debug info:
> > > gstbasesrc.c(3055): gst_base_src_loop ():
> > /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
> > > streaming stopped, reason error (-5)
> > 
> > Looks like the real error was not propagated (let me know which version you
> > are
> > using, Ubuntu tends to be far behind, or only cherry-picking fixes rather
> > then
> > tracking stable branches).
> > 
> > To find our more, set GST_DEBUG="v4l2*:7" env. That will give you a lot more
> > details. By experience, I suspect that your output-io-mode=dmabuf-import is
> > faulty on the v4l2convert element (on newer GStreamer, we now have a fix
> > name
> > for converter too). 
> > 
> > IMX.6 platform (and IMX.8 too) didn't include an IOMMU in their platform. On
> > the
> > other side, USB cameras (like UVC) produces scattered memory (virtual
> > memory).
> > That makes importation of dmabuf from UVC to v4l2convert impossible in this
> > case. You may want to try the other way around, to set io-mode=dmabuf-import 
> > on
> > the v4l2src element. If that does not work, you'll have no choice but to let
> > the
> > pipelein makes a copy (default setting). Normally, I suggest to first try
> > without touching these advance settings, to make it work, and then tweak to
> > gain
> > more performance.
> > 
> > > Execution ended after 0:00:00.436501205
> > > Setting pipeline to PAUSED ...
> > > Setting pipeline to READY ...
> > > Setting pipeline to NULL ...
> > > Freeing pipeline ...
> > > 
> > > Any hint on debugging would be appreciated!
> > > Thank you,
> > >   -- Constantine.
> > > 
> > > 
> > > 
> > > 
> > > 
> > > On Sun, Jan 19, 2020 at 2:13 PM Milian Wolff <milian.wolff at kdab.com>
> > wrote:
> > > > On Sonntag, 19. Januar 2020 11:06:55 CET Constantine Elster wrote:
> > > > > Hi devs,
> > > > > 
> > > > > I'm trying to construct a pipeline that captures frames from a USB
> > camera
> > > > > (YUV) and encodes them with HW encoder and saves into a file. My setup
> > is
> > > > > iMX6 board running Ubuntu 18.04 on 4.20 mainline kernel.
> > > > > 
> > > > > When I try a sw encoder, it works okay albeit I get very high 100% CPU
> > > > > usage. The working pipeline based on software plugins:
> > > > > gst-launch-1.0 -v v4l2src device="/dev/video2" num-buffers=200 !
> > > > > "video/x-raw, format=(string)UYVY, width=(int)640, height=(int)480,
> > > > > pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)45/1,
> > > > > colorimetry=bt709" ! videoconvert ! x264enc ! mp4mux ! filesink
> > > > > location=aha.mp4
> > > > > 
> > > > > My attempt to replace sw based plugins by HW based with efficient
> > memory
> > > > > management:
> > > > > gst-launch-1.0 -v v4l2src device="/dev/video9" num-buffers=200 !
> > > > > "video/x-raw, format=(string)UYVY, width=(int)640, height=(int)480,
> > > > > pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)45/1,
> > > > > colorimetry=bt709" ! *v4l2video8convert* *output-io-mode=dmabuf-
> > import* !
> > > > > *v4l2h264enc* *output-io-mode=dmabuf-import* ! mp4mux ! filesink
> > > > > location=aha.mp4
> > > > > 
> > > > > I get the following error: "WARNING: erroneous pipeline: could not
> > link
> > > > > v4l2h264enc0 to mp4mux0"
> > > > > 
> > > > > Would appreciate any ideas how to understand what's wrong, how to
> > debug and
> > > > > make it work.
> > > > 
> > > > To debug, I suggest you compare the SRC of `gst-inspect-1.0 v4l2h264enc`
> > with 
> > > > the SINK of `gst-inspect-1.0 mp4mux`. My guess is that you may be able
> > to fix 
> > > > the issue by adding a `h264parse` element in the middle to fix the
> > alignment 
> > > > since `mp4mux` requires `au` alignment, whereas the encoder may output
> > `nal` 
> > > > frames?
> > > > 
> > > > Good luck
> > > > 
> > > > -- 
> > > > Milian Wolff | milian.wolff at kdab.com | Senior Software Engineer
> > > > KDAB (Deutschland) GmbH, a KDAB Group company
> > > > Tel: +49-30-521325470
> > > > KDAB - The Qt, C++ and OpenGL Experts
> > > 
> > > 
> > > _______________________________________________
> > > gstreamer-devel mailing list
> > > gstreamer-devel at lists.freedesktop.org
> > > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> > 
> > _______________________________________________
> > gstreamer-devel mailing list
> > gstreamer-devel at lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> 
> 
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel