issues with v4l2convert usage for RGB -> YUV conversion
Nicolas Dufresne
nicolas at ndufresne.ca
Fri Apr 24 14:48:38 UTC 2020
Le vendredi 24 avril 2020 à 12:14 +0200, Milian Wolff a écrit :
> Hey there!
>
> this is a re-post of [1] to this mailing list, as I was directed here.
>
> [1]: https://www.raspberrypi.org/forums/viewtopic.php?
> f=70&t=271720&sid=c5b8e22d61541521d6509330663b5fa8
>
> We are trying to optimize a gstreamer pipeline running on a rpi3b+ where,
> according to gst-shark, the current main bottleneck is a videoconvert element.
> That one is necessary to convert OpenGL frames in RGBA format to YUV for
> omxh264enc. This is a simplified example pipeline:
>
> ```
> gst-launch-1.0 gltestsrc \
> ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
> ! gldownload ! queue \
> ! videoconvert \
> ! omxh264enc ! avimux ! filesink location=test.avi
> ```
>
> My first approach was trying to get omxh264enc work directly with RGBA frames,
> see: https://gitlab.freedesktop.org/gstreamer/gst-omx/-/merge_requests/62
>
> This in turn led me to v4l2convert and v4l2h264enc (see https://
> www.raspberrypi.org/forums/viewtopic.php?t=245852). We have now updated our
> stack to the following versions:
>
> ```
> linux 4.19.88
> gstreamer 1.16.1
> ```
>
> When I replace the omxh264enc with v4l2h264enc I run into the first issue:
>
> ```
> gst-launch-1.0 gltestsrc \
> ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
> ! gldownload ! queue \
> ! videoconvert \
> ! v4l2h264enc ! avimux ! filesink location=test.avi
>
> ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: Internal
> data stream error.
> Additional debug info:
> ../../../../gstreamer-1.16.1/libs/gst/base/gstbasesrc.c(3072):
> gst_base_src_loop (): /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0:
> streaming stopped, reason not-negotiated (-4)
> ```
>
> Can someone explain what's going on here? I've also tried adding a
> `glcolorconvert ! "video/x-raw(memory:GLMemory),format=BGRA"` before the
> gldownload, but to no avail.
>
> The full output with GST_DEBUG="*:3" is:
>
> ```
> Setting pipeline to PAUSED ...
> 0:00:00.238494323 1765 0x2361ba0 WARN v4l2
> gstv4l2object.c:4196:gst_v4l2_object_probe_caps:<v4l2h264enc0:src> Failed to
> probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
> Pipeline is PREROLLING ...
> 0:00:00.242415365 1765 0x23386c0 FIXME default gstutils.c:
> 3981:gst_pad_create_stream_id_internal:<gltestsrc0:src> Creating random
> stream-id, consider implementing a deterministic way of creating a stream-id
> Got context from element 'gldownloadelement0': gst.gl.GLDisplay=context,
> gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0";
> 0:00:00.264218855 1765 0x23381b0 WARN basetransform
> gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0> transform
> could not transform video/x-raw, format=(string)RGBA, width=(int)640,
> height=(int)480, framerate=(fraction)30/1, interlace-mode=(string)progressive
> in anything we support
> 0:00:00.271333542 1765 0x23381b0 WARN basetransform
> gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0> transform
> could not transform video/x-raw, format=(string)RGBA, width=(int)640,
> height=(int)480, framerate=(fraction)30/1, interlace-mode=(string)progressive
> in anything we support
> 0:00:00.271435157 1765 0x23381b0 WARN GST_PADS gstpad.c:
> 4231:gst_pad_peer_query:<queue0:src> could not send sticky events
> 0:00:00.293517917 1765 0x23581b0 FIXME glslstage
> gstglslstage.c:518:_compile_shader:<glslstage0> vertex shader info
> log:Compiled
> 0:00:00.294581980 1765 0x23581b0 FIXME glslstage
> gstglslstage.c:518:_compile_shader:<glslstage1> fragment shader info
> log:Compiled
> 0:00:00.306531615 1765 0x23581b0 FIXME glslstage
> gstglslstage.c:518:_compile_shader:<glslstage2> vertex shader info
> log:Compiled
> 0:00:00.307533698 1765 0x23581b0 FIXME glslstage
> gstglslstage.c:518:_compile_shader:<glslstage3> fragment shader info
> log:Compiled
> 0:00:00.342931927 1765 0x23381b0 WARN basetransform
> gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0> transform
> could not transform video/x-raw, format=(string)RGBA, width=(int)640,
> height=(int)480, framerate=(fraction)30/1, interlace-mode=(string)progressive
> in anything we support
> 0:00:00.350237917 1765 0x23381b0 WARN basetransform
> gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0> transform
> could not transform video/x-raw, format=(string)RGBA, width=(int)640,
> height=(int)480, framerate=(fraction)30/1, interlace-mode=(string)progressive
> in anything we support
> 0:00:00.354808438 1765 0x23386c0 WARN basesrc gstbasesrc.c:
> 3072:gst_base_src_loop:<gltestsrc0> error: Internal data stream error.
> 0:00:00.355374011 1765 0x23386c0 WARN basesrc gstbasesrc.c:
> 3072:gst_base_src_loop:<gltestsrc0> error: streaming stopped, reason not-
> negotiated (-4)
> 0:00:00.356358646 1765 0x23386c0 WARN queue gstqueue.c:
> 988:gst_queue_handle_sink_event:<queue0> error: Internal data stream error.
> ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: Internal
> data stream error.
> Additional debug info:
> ../../../../gstreamer-1.16.1/libs/gst/base/gstbasesrc.c(3072):
> gst_base_src_loop (): /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0:
> streaming stopped, reason not-negotiated (-4)
> ERROR: pipeline doesn't want to preroll.
> Setting pipeline to NULL ...
> 0:00:00.357002709 1765 0x23386c0 WARN queue gstqueue.c:
> 988:gst_queue_handle_sink_event:<queue0> error: streaming stopped, reason not-
> negotiated (-4)
> Freeing pipeline ...
> ```
>
> I can manage to workaround this by explicitly letting the videoconvert do the
> YUV conversion like this:
>
> ```
> gst-launch-1.0 gltestsrc \
> ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
> ! gldownload ! queue \
> ! videoconvert ! "video/x-raw,format=I420" \
> ! v4l2h264enc ! avimux ! filesink location=test.avi
> ```
>
> No more errors, but the resulting test.avi file is completely broken. I can
> play it with cvlc but its all green and garbled... Can someone explain what's
> going on here?
>
> Now, let's try to stick to omxh264enc and replace videoconvert instead:
>
> ```
> gst-launch-1.0 gltestsrc \
> ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
> ! gldownload ! queue \
> ! v4l2convert \
> ! omxh264enc ! avimux ! filesink location=test.avi
>
> WARNING: erroneous pipeline: could not link queue0 to v4l2convert0
As the driver author mention, they converter driver does not support
RGBA (it's such a rare format afaic). I would suggest you to introduce
glcolorconvert element right before gldownload. It's even likely that
the converter won't be required afterward, as the GPU will take care of
the color conversion.
> ```
>
> Which is odd, but expected considering the following:
>
> ```
> gst-inspect-1.0 v4l2convert
> ...
> video/x-raw
> format: { (string)YUY2, (string)YVYU, (string)UYVY,
> (string)I420, (string)YV12, (string)RGB, (string)BGR, (string)BGR
> x, (string)BGRA, (string)RGB16, (string)NV12, (string)NV21 }
> width: [ 1, 32768 ]
> height: [ 1, 32768 ]
> framerate: [ 0/1, 2147483647/1 ]
> ```
>
> So, why is RGBA not supported here? Well, let's try to add a glcolorconvert to
> BGRA, but that fails too:
>
> ```
> gst-launch-1.0 gltestsrc \
> ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \
> ! glcolorconvert ! "video/x-raw(memory:GLMemory),format=BGRA" \
> ! gldownload ! queue \
> ! v4l2convert \
> ! omxh264enc ! avimux ! filesink location=test.avi
>
> ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: Internal
> data stream error.
> Additional debug info:
> ../../../../gstreamer-1.16.1/libs/gst/base/gstbasesrc.c(3072):
> gst_base_src_loop (): /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0:
> streaming stopped, reason not-negotiated (-4)
> ERROR: pipeline doesn't want to preroll.
> ```
>
> I've tried some of the other RGB formats, but none seem to work here. What am
> I missing? I've attached the full `:4 GST_DEBUG trace log` file over on the
> rpi forum at [2]
>
> [2]: https://www.raspberrypi.org/forums/download/file.php?
> id=36357&sid=376f023986c765adc4950fe5e1335652
>
> Thanks a lot
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
More information about the gstreamer-devel
mailing list