<div dir="ltr">i use RGBA to pack gray16 into an efficient RTP approved format. <div><a href="https://wiki.idiot.io/raw">https://wiki.idiot.io/raw</a> </div></div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 24, 2020 at 5:48 PM Nicolas Dufresne <<a href="mailto:nicolas@ndufresne.ca">nicolas@ndufresne.ca</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Le vendredi 24 avril 2020 à 12:14 +0200, Milian Wolff a écrit : > Hey there! > > this is a re-post of [1] to this mailing list, as I was directed here. > > [1]: <a href="https://www.raspberrypi.org/forums/viewtopic.php" rel="noreferrer" target="_blank">https://www.raspberrypi.org/forums/viewtopic.php</a>? > f=70&t=271720&sid=c5b8e22d61541521d6509330663b5fa8 > > We are trying to optimize a gstreamer pipeline running on a rpi3b+ where, > according to gst-shark, the current main bottleneck is a videoconvert element. > That one is necessary to convert OpenGL frames in RGBA format to YUV for > omxh264enc. This is a simplified example pipeline: > > ``` > gst-launch-1.0 gltestsrc \ > ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \ > ! gldownload ! queue \ > ! videoconvert \ > ! omxh264enc ! avimux ! filesink location=test.avi > ``` > > My first approach was trying to get omxh264enc work directly with RGBA frames, > see: <a href="https://gitlab.freedesktop.org/gstreamer/gst-omx/-/merge_requests/62" rel="noreferrer" target="_blank">https://gitlab.freedesktop.org/gstreamer/gst-omx/-/merge_requests/62</a> > > This in turn led me to v4l2convert and v4l2h264enc (see https:// > <a href="http://www.raspberrypi.org/forums/viewtopic.php?t=245852" rel="noreferrer" target="_blank">www.raspberrypi.org/forums/viewtopic.php?t=245852</a>). We have now updated our > stack to the following versions: > > ``` > linux 4.19.88 > gstreamer 1.16.1 > ``` > > When I replace the omxh264enc with v4l2h264enc I run into the first issue: > > ``` > gst-launch-1.0 gltestsrc \ > ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \ > ! gldownload ! queue \ > ! videoconvert \ > ! v4l2h264enc ! avimux ! filesink location=test.avi > > ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: Internal > data stream error. > Additional debug info: > ../../../../gstreamer-1.16.1/libs/gst/base/gstbasesrc.c(3072): > gst_base_src_loop (): /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: > streaming stopped, reason not-negotiated (-4) > ``` > > Can someone explain what's going on here? I've also tried adding a > `glcolorconvert ! "video/x-raw(memory:GLMemory),format=BGRA"` before the > gldownload, but to no avail. > > The full output with GST_DEBUG="*:3" is: > > ``` > Setting pipeline to PAUSED ... > 0:00:00.238494323 1765 0x2361ba0 WARN v4l2 > gstv4l2object.c:4196:gst_v4l2_object_probe_caps:<v4l2h264enc0:src> Failed to > probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument > Pipeline is PREROLLING ... > 0:00:00.242415365 1765 0x23386c0 FIXME default gstutils.c: > 3981:gst_pad_create_stream_id_internal:<gltestsrc0:src> Creating random > stream-id, consider implementing a deterministic way of creating a stream-id > Got context from element 'gldownloadelement0': gst.gl.GLDisplay=context, > gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0"; > 0:00:00.264218855 1765 0x23381b0 WARN basetransform > gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0> transform > could not transform video/x-raw, format=(string)RGBA, width=(int)640, > height=(int)480, framerate=(fraction)30/1, interlace-mode=(string)progressive > in anything we support > 0:00:00.271333542 1765 0x23381b0 WARN basetransform > gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0> transform > could not transform video/x-raw, format=(string)RGBA, width=(int)640, > height=(int)480, framerate=(fraction)30/1, interlace-mode=(string)progressive > in anything we support > 0:00:00.271435157 1765 0x23381b0 WARN GST_PADS gstpad.c: > 4231:gst_pad_peer_query:<queue0:src> could not send sticky events > 0:00:00.293517917 1765 0x23581b0 FIXME glslstage > gstglslstage.c:518:_compile_shader:<glslstage0> vertex shader info > log:Compiled > 0:00:00.294581980 1765 0x23581b0 FIXME glslstage > gstglslstage.c:518:_compile_shader:<glslstage1> fragment shader info > log:Compiled > 0:00:00.306531615 1765 0x23581b0 FIXME glslstage > gstglslstage.c:518:_compile_shader:<glslstage2> vertex shader info > log:Compiled > 0:00:00.307533698 1765 0x23581b0 FIXME glslstage > gstglslstage.c:518:_compile_shader:<glslstage3> fragment shader info > log:Compiled > 0:00:00.342931927 1765 0x23381b0 WARN basetransform > gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0> transform > could not transform video/x-raw, format=(string)RGBA, width=(int)640, > height=(int)480, framerate=(fraction)30/1, interlace-mode=(string)progressive > in anything we support > 0:00:00.350237917 1765 0x23381b0 WARN basetransform > gstbasetransform.c:1364:gst_base_transform_setcaps:<videoconvert0> transform > could not transform video/x-raw, format=(string)RGBA, width=(int)640, > height=(int)480, framerate=(fraction)30/1, interlace-mode=(string)progressive > in anything we support > 0:00:00.354808438 1765 0x23386c0 WARN basesrc gstbasesrc.c: > 3072:gst_base_src_loop:<gltestsrc0> error: Internal data stream error. > 0:00:00.355374011 1765 0x23386c0 WARN basesrc gstbasesrc.c: > 3072:gst_base_src_loop:<gltestsrc0> error: streaming stopped, reason not- > negotiated (-4) > 0:00:00.356358646 1765 0x23386c0 WARN queue gstqueue.c: > 988:gst_queue_handle_sink_event:<queue0> error: Internal data stream error. > ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: Internal > data stream error. > Additional debug info: > ../../../../gstreamer-1.16.1/libs/gst/base/gstbasesrc.c(3072): > gst_base_src_loop (): /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: > streaming stopped, reason not-negotiated (-4) > ERROR: pipeline doesn't want to preroll. > Setting pipeline to NULL ... > 0:00:00.357002709 1765 0x23386c0 WARN queue gstqueue.c: > 988:gst_queue_handle_sink_event:<queue0> error: streaming stopped, reason not- > negotiated (-4) > Freeing pipeline ... > ``` > > I can manage to workaround this by explicitly letting the videoconvert do the > YUV conversion like this: > > ``` > gst-launch-1.0 gltestsrc \ > ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \ > ! gldownload ! queue \ > ! videoconvert ! "video/x-raw,format=I420" \ > ! v4l2h264enc ! avimux ! filesink location=test.avi > ``` > > No more errors, but the resulting test.avi file is completely broken. I can > play it with cvlc but its all green and garbled... Can someone explain what's > going on here? > > Now, let's try to stick to omxh264enc and replace videoconvert instead: > > ``` > gst-launch-1.0 gltestsrc \ > ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \ > ! gldownload ! queue \ > ! v4l2convert \ > ! omxh264enc ! avimux ! filesink location=test.avi > > WARNING: erroneous pipeline: could not link queue0 to v4l2convert0 As the driver author mention, they converter driver does not support RGBA (it's such a rare format afaic). I would suggest you to introduce glcolorconvert element right before gldownload. It's even likely that the converter won't be required afterward, as the GPU will take care of the color conversion. > ``` > > Which is odd, but expected considering the following: > > ``` > gst-inspect-1.0 v4l2convert > ... > video/x-raw > format: { (string)YUY2, (string)YVYU, (string)UYVY, > (string)I420, (string)YV12, (string)RGB, (string)BGR, (string)BGR > x, (string)BGRA, (string)RGB16, (string)NV12, (string)NV21 } > width: [ 1, 32768 ] > height: [ 1, 32768 ] > framerate: [ 0/1, 2147483647/1 ] > ``` > > So, why is RGBA not supported here? Well, let's try to add a glcolorconvert to > BGRA, but that fails too: > > ``` > gst-launch-1.0 gltestsrc \ > ! "video/x-raw(memory:GLMemory),format=RGBA,width=640,height=480" \ > ! glcolorconvert ! "video/x-raw(memory:GLMemory),format=BGRA" \ > ! gldownload ! queue \ > ! v4l2convert \ > ! omxh264enc ! avimux ! filesink location=test.avi > > ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: Internal > data stream error. > Additional debug info: > ../../../../gstreamer-1.16.1/libs/gst/base/gstbasesrc.c(3072): > gst_base_src_loop (): /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: > streaming stopped, reason not-negotiated (-4) > ERROR: pipeline doesn't want to preroll. > ``` > > I've tried some of the other RGB formats, but none seem to work here. What am > I missing? I've attached the full `:4 GST_DEBUG trace log` file over on the > rpi forum at [2] > > [2]: <a href="https://www.raspberrypi.org/forums/download/file.php" rel="noreferrer" target="_blank">https://www.raspberrypi.org/forums/download/file.php</a>? > id=36357&sid=376f023986c765adc4950fe5e1335652 > > Thanks a lot > _______________________________________________ > gstreamer-devel mailing list > <a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a> > <a href="https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel" rel="noreferrer" target="_blank">https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel</a> _______________________________________________ gstreamer-devel mailing list <a href="mailto:gstreamer-devel@lists.freedesktop.org" target="_blank">gstreamer-devel@lists.freedesktop.org</a> <a href="https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel" rel="noreferrer" target="_blank">https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel</a> </blockquote></div>