issues with v4l2convert usage for RGB -> YUV conversion
Milian Wolff
milian.wolff at kdab.com
Sun Apr 26 20:04:31 UTC 2020
On Freitag, 24. April 2020 14:26:18 CEST Yair Reshef wrote:
> You should consider converting to yuv on the opengl side. Saving transfer
> penelty to/from gpu.
>
> Nvidia has a strain of gstreamer that handles converting on the gpu. For
> Jetson platform.
> Also ridgerun is active there with gstisp, not sure about raspi support.
The Raspberry Pi 3b+ GPU does not have the necessary capabilities for this:
> 0:00:00.370688802 314 0x71602c30 ERROR glconvert
> gstglcolorconvert.c:2135:_init_convert: Conversion requires output to
> multiple draw buffers
> 0:00:00.370827969 314 0x71602c30 ERROR
> glconvert gstglcolorconvert.c:2247:_init_convert:<glcolorconvert0>
> Converting from RGBA to I420 requires functionality that the current OpenGL
> setup does not support 0:00:00.370939792 314 0x7c5860 WARN
> glconvertelement
> gstglcolorconvertelement.c:215:gst_gl_color_convert_element_prepare_output_
> buffer:<glcolorconvertelement0> error: Failed to convert video buffer
>From the code:
> multiple draw targets not supported on GLES2...yet
Otherwise yes I think that would be the ideal solution...
> On Fri, Apr 24, 2020, 13:15 Milian Wolff <milian.wolff at kdab.com> wrote:
> > Hey there!
> >
> > this is a re-post of [1] to this mailing list, as I was directed here.
> >
> > [1]: https://www.raspberrypi.org/forums/viewtopic.php?
> > f=70&t=271720&sid=c5b8e22d61541521d6509330663b5fa8
> >
> > We are trying to optimize a gstreamer pipeline running on a rpi3b+ where,
> > according to gst-shark, the current main bottleneck is a videoconvert
> > element.
> > That one is necessary to convert OpenGL frames in RGBA format to YUV for
> > omxh264enc. This is a simplified example pipeline:
> >
> > ```
> > gst-launch-1.0 gltestsrc \
> >
> > ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
> > ! gldownload ! queue \
> > ! videoconvert \
> > ! omxh264enc ! avimux ! filesink location=test.avi
> >
> > ```
> >
> > My first approach was trying to get omxh264enc work directly with RGBA
> > frames,
> > see: https://gitlab.freedesktop.org/gstreamer/gst-omx/-/merge_requests/62
> >
> > This in turn led me to v4l2convert and v4l2h264enc (see https://
> > www.raspberrypi.org/forums/viewtopic.php?t=245852). We have now updated
> > our
> > stack to the following versions:
> >
> > ```
> > linux 4.19.88
> > gstreamer 1.16.1
> > ```
> >
> > When I replace the omxh264enc with v4l2h264enc I run into the first issue:
> >
> > ```
> > gst-launch-1.0 gltestsrc \
> >
> > ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
> > ! gldownload ! queue \
> > ! videoconvert \
> > ! v4l2h264enc ! avimux ! filesink location=test.avi
> >
> > ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0:
> > Internal
> > data stream error.
> > Additional debug info:
> > ../../../../gstreamer-1.16.1/libs/gst/base/gstbasesrc.c(3072):
> > gst_base_src_loop (): /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0:
> > streaming stopped, reason not-negotiated (-4)
> > ```
> >
> > Can someone explain what's going on here? I've also tried adding a
> > `glcolorconvert ! "video/x-raw(memory:GLMemory),format=BGRA"` before the
> > gldownload, but to no avail.
> >
> > The full output with GST_DEBUG="*:3" is:
> >
> > ```
> > Setting pipeline to PAUSED ...
> > 0:00:00.238494323 1765 0x2361ba0 WARN v4l2
> > gstv4l2object.c:4196:gst_v4l2_object_probe_caps:<v4l2h264enc0:src> Failed
> > to
> > probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
> > Pipeline is PREROLLING ...
> > 0:00:00.242415365 1765 0x23386c0 FIXME default
> > gstutils.c:
> > 3981:gst_pad_create_stream_id_internal:<gltestsrc0:src> Creating random
> > stream-id, consider implementing a deterministic way of creating a
> > stream-id
> > Got context from element 'gldownloadelement0': gst.gl.GLDisplay=context,
> > gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0";
> > 0:00:00.264218855 1765 0x23381b0 WARN basetransform
> > gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0>
> > transform
> > could not transform video/x-raw, format=(string)RGBA, width=(int)640,
> > height=(int)480, framerate=(fraction)30/1,
> > interlace-mode=(string)progressive
> > in anything we support
> > 0:00:00.271333542 1765 0x23381b0 WARN basetransform
> > gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0>
> > transform
> > could not transform video/x-raw, format=(string)RGBA, width=(int)640,
> > height=(int)480, framerate=(fraction)30/1,
> > interlace-mode=(string)progressive
> > in anything we support
> > 0:00:00.271435157 1765 0x23381b0 WARN GST_PADS gstpad.c:
> > 4231:gst_pad_peer_query:<queue0:src> could not send sticky events
> > 0:00:00.293517917 1765 0x23581b0 FIXME glslstage
> > gstglslstage.c:518:_compile_shader:<glslstage0> vertex shader info
> > log:Compiled
> > 0:00:00.294581980 1765 0x23581b0 FIXME glslstage
> > gstglslstage.c:518:_compile_shader:<glslstage1> fragment shader info
> > log:Compiled
> > 0:00:00.306531615 1765 0x23581b0 FIXME glslstage
> > gstglslstage.c:518:_compile_shader:<glslstage2> vertex shader info
> > log:Compiled
> > 0:00:00.307533698 1765 0x23581b0 FIXME glslstage
> > gstglslstage.c:518:_compile_shader:<glslstage3> fragment shader info
> > log:Compiled
> > 0:00:00.342931927 1765 0x23381b0 WARN basetransform
> > gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0>
> > transform
> > could not transform video/x-raw, format=(string)RGBA, width=(int)640,
> > height=(int)480, framerate=(fraction)30/1,
> > interlace-mode=(string)progressive
> > in anything we support
> > 0:00:00.350237917 1765 0x23381b0 WARN basetransform
> > gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0>
> > transform
> > could not transform video/x-raw, format=(string)RGBA, width=(int)640,
> > height=(int)480, framerate=(fraction)30/1,
> > interlace-mode=(string)progressive
> > in anything we support
> > 0:00:00.354808438 1765 0x23386c0 WARN basesrc
> > gstbasesrc.c:
> > 3072:gst_base_src_loop:<gltestsrc0> error: Internal data stream error.
> > 0:00:00.355374011 1765 0x23386c0 WARN basesrc
> > gstbasesrc.c:
> > 3072:gst_base_src_loop:<gltestsrc0> error: streaming stopped, reason not-
> > negotiated (-4)
> > 0:00:00.356358646 1765 0x23386c0 WARN queue
> > gstqueue.c:
> > 988:gst_queue_handle_sink_event:<queue0> error: Internal data stream
> > error.
> > ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0:
> > Internal
> > data stream error.
> > Additional debug info:
> > ../../../../gstreamer-1.16.1/libs/gst/base/gstbasesrc.c(3072):
> > gst_base_src_loop (): /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0:
> > streaming stopped, reason not-negotiated (-4)
> > ERROR: pipeline doesn't want to preroll.
> > Setting pipeline to NULL ...
> > 0:00:00.357002709 1765 0x23386c0 WARN queue
> > gstqueue.c:
> > 988:gst_queue_handle_sink_event:<queue0> error: streaming stopped, reason
> > not-
> > negotiated (-4)
> > Freeing pipeline ...
> > ```
> >
> > I can manage to workaround this by explicitly letting the videoconvert do
> > the
> > YUV conversion like this:
> >
> > ```
> > gst-launch-1.0 gltestsrc \
> >
> > ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
> > ! gldownload ! queue \
> > ! videoconvert ! "video/x-raw,format=I420" \
> > ! v4l2h264enc ! avimux ! filesink location=test.avi
> >
> > ```
> >
> > No more errors, but the resulting test.avi file is completely broken. I
> > can
> > play it with cvlc but its all green and garbled... Can someone explain
> > what's
> > going on here?
> >
> > Now, let's try to stick to omxh264enc and replace videoconvert instead:
> >
> > ```
> > gst-launch-1.0 gltestsrc \
> >
> > ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
> > ! gldownload ! queue \
> > ! v4l2convert \
> > ! omxh264enc ! avimux ! filesink location=test.avi
> >
> > WARNING: erroneous pipeline: could not link queue0 to v4l2convert0
> > ```
> >
> > Which is odd, but expected considering the following:
> >
> > ```
> > gst-inspect-1.0 v4l2convert
> > ...
> >
> > video/x-raw
> >
> > format: { (string)YUY2, (string)YVYU, (string)UYVY,
> >
> > (string)I420, (string)YV12, (string)RGB, (string)BGR, (string)BGR
> > x, (string)BGRA, (string)RGB16, (string)NV12, (string)NV21 }
> >
> > width: [ 1, 32768 ]
> >
> > height: [ 1, 32768 ]
> >
> > framerate: [ 0/1, 2147483647/1 ]
> >
> > ```
> >
> > So, why is RGBA not supported here? Well, let's try to add a
> > glcolorconvert to
> > BGRA, but that fails too:
> >
> > ```
> > gst-launch-1.0 gltestsrc \
> >
> > ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
> > ! glcolorconvert ! "video/x-raw(memory:GLMemory),format=BGRA" \
> > ! gldownload ! queue \
> > ! v4l2convert \
> > ! omxh264enc ! avimux ! filesink location=test.avi
> >
> > ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0:
> > Internal
> > data stream error.
> > Additional debug info:
> > ../../../../gstreamer-1.16.1/libs/gst/base/gstbasesrc.c(3072):
> > gst_base_src_loop (): /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0:
> > streaming stopped, reason not-negotiated (-4)
> > ERROR: pipeline doesn't want to preroll.
> > ```
> >
> > I've tried some of the other RGB formats, but none seem to work here. What
> > am
> > I missing? I've attached the full `:4 GST_DEBUG trace log` file over on
> > the
> > rpi forum at [2]
> >
> > [2]: https://www.raspberrypi.org/forums/download/file.php?
> > id=36357&sid=376f023986c765adc4950fe5e1335652
> >
> > Thanks a lot
> > --
> > Milian Wolff | milian.wolff at kdab.com | Senior Software Engineer
> > KDAB (Deutschland) GmbH, a KDAB Group company
> > Tel: +49-30-521325470
> > KDAB - The Qt, C++ and OpenGL
> > Experts_______________________________________________
> > gstreamer-devel mailing list
> > gstreamer-devel at lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
--
Milian Wolff | milian.wolff at kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3826 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20200426/4d5ba2cb/attachment-0001.bin>
More information about the gstreamer-devel
mailing list