What's the meaning of video/x-raw(memory:GLMemory) without texture-target?
Milian Wolff
milian.wolff at kdab.com
Mon Nov 25 10:31:18 UTC 2019
On Monday, November 25, 2019 11:20:00 AM CET Matthew Waters wrote:
> Try adding a glupload ! glcolorconvert before the gleffects_twirl
> element. glimagesink contains these automatically already.
Thanks! This fixes the errors but it leaves me equally puzzled:
Most notably, the twirl effect isn't taking effect, i.e. this is my command
and debug output now:
```
# GST_DEBUG=3 gst-launch-1.0 rpicamsrc ! "video/x-h264,width=640,height=480" !
h264parse ! omxh264dec ! "video/x-raw(memory:GLMemory)" ! glupload !
gleffects_twirl ! queue ! glimagesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
0:00:00.489548698 381 0x1c45af0 FIXME default gstutils.c:
3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random
stream-id, consider implementing a deterministic way of creating a stream-id
Got context from element 'sink': gst.gl.GLDisplay=context,
gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0";
Setting pipeline to PLAYING ...
New clock: GstSystemClock
0:00:00.880536979 381 0x1c45af0 ERROR omxvideodec
gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0>
Empty caps
0:00:01.057018906 381 0x1c2f750 FIXME glslstage
gstglslstage.c:518:_compile_shader:<glslstage0> vertex shader info
log:Compiled
0:00:01.067645364 381 0x1c2f750 FIXME glslstage
gstglslstage.c:518:_compile_shader:<glslstage1> fragment shader info
log:Compiled
0:00:01.125377187 381 0x1c2f750 FIXME glslstage
gstglslstage.c:518:_compile_shader:<glslstage2> vertex shader info
log:Compiled
0:00:01.127863541 381 0x1c2f750 FIXME glslstage
gstglslstage.c:518:_compile_shader:<glslstage3> fragment shader info
log:Compiled
0:00:01.160143698 381 0x1c2f750 FIXME glslstage
gstglslstage.c:518:_compile_shader:<glslstage4> vertex shader info
log:Compiled
0:00:01.161840260 381 0x1c2f750 FIXME glslstage
gstglslstage.c:518:_compile_shader:<glslstage5> fragment shader info
log:Compiled
```
But the camera image is displayed as-is. What's odd is that I seem to be
seeing the twirl effect for a very short time when I "Ctrl+C" terminate the
`gst-launch` process.
If I use `gltestsrc` as a replacement for the source up to `omxh264dec`, I see
the twirl effect directly. What's going on here?
Secondly, could you please explain to me why the `glupload ! glcolorconvert`
is required here? Most notably, `video/x-raw(GLMemory)` should be on the GPU
already, why do I need to "upload" it again? And the format is already `RGBA`,
so why do I need glcolorconvert?
Thanks
> On 25/11/19 8:30 pm, Milian Wolff wrote:
> > On Friday, November 22, 2019 5:12:57 AM CET Matthew Waters wrote:
> >> Hi,
> >
> > Hey Matt!
> >
> >> By default a 'video/x-raw(memory:GLMemory)' caps without the
> >> texture-target field is equivalent to texture-target=2D as that was the
> >> default before we differentiated the different OpenGL texture targets.
> >> Now elements should be able to negotiate the differences between them
> >> correctly however there may be bugs.
> >
> > OK, thanks - good to know!
> >
> >> The GLMemory output from omxh264dec is highly dependant on the platform
> >> you are running on and may or may not work correctly. I know that it
> >> used to work on the RPi and have not tested any other omx platform
> >> myself.
> >
> > What's interesting is that this seems to work fine when I shove the data
> > directly into the glimagesink, i.e.:
> >
> > ```
> > # GST_DEBUG=3 gst-launch-1.0 rpicamsrc !
> > "video/x-h264,width=640,height=480" ! h264parse ! omxh264dec !
> > "video/x-raw(memory:GLMemory)" ! queue !
> > glimagesinkSetting pipeline to PAUSED ...
> > Pipeline is live and does not need PREROLL ...
> > 0:00:00.488648854 31116 0x167e7b0 FIXME default
> > gstutils.c:
> > 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random
> > stream-id, consider implementing a deterministic way of creating a
> > stream-id Got context from element 'sink': gst.gl.GLDisplay=context,
> > gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0";
> > Setting pipeline to PLAYING ...
> > New clock: GstSystemClock
> > 0:00:00.894124323 31116 0x167e7b0 ERROR omxvideodec
> > gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0>
> > Empty caps
> > 0:00:00.985837604 31116 0x167e230 FIXME glslstage
> > gstglslstage.c:518:_compile_shader:<glslstage0> vertex shader info
> > log:Compiled
> > 0:00:00.990119375 31116 0x167e230 FIXME glslstage
> > gstglslstage.c:518:_compile_shader:<glslstage1> fragment shader info
> > log:Compiled
> > 0:00:01.013341718 31116 0x167e230 FIXME glslstage
> > gstglslstage.c:518:_compile_shader:<glslstage2> vertex shader info
> > log:Compiled
> > 0:00:01.014671666 31116 0x167e230 FIXME glslstage
> > gstglslstage.c:518:_compile_shader:<glslstage3> fragment shader info
> > log:Compiled
> > ^Chandling interrupt.
> > Interrupt: Stopping pipeline ...
> > Execution ended after 0:00:06.801416612
> > Setting pipeline to PAUSED ...
> > Setting pipeline to READY ...
> > Setting pipeline to NULL ...
> > Freeing pipeline ...
> > ```
> >
> > I don't know what the ERROR is trying to tell me, but this pipeline works
> > a
> > charm.
> >
> >> The first step to debugging this would be to enable GStreamer debug
> >> logging to get an idea of what may be wrong.
> >
> > But the following happens when I add the gleffects_twirl filter (either
> > before or after the queue, doesn't matter):
> >
> > ```
> > # GST_DEBUG=3 gst-launch-1.0 rpicamsrc !
> > "video/x-h264,width=640,height=480" ! h264parse ! omxh264dec !
> > "video/x-raw(memory:GLMemory)" ! gleffects_twirl ! queue ! glimagesink
> > Setting pipeline to PAUSED ...
> > Pipeline is live and does not need PREROLL ...
> > 0:00:00.506736041 31143 0x1fed490 FIXME default
> > gstutils.c:
> > 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random
> > stream-id, consider implementing a deterministic way of creating a
> > stream-id Got context from element 'sink': gst.gl.GLDisplay=context,
> > gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0";
> > Setting pipeline to PLAYING ...
> > New clock: GstSystemClock
> > 0:00:00.882512708 31143 0x1fed490 ERROR omxvideodec
> > gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0>
> > Empty caps
> > 0:00:00.968698020 31143 0x6fd03290 WARN videodecoder
> > gstvideodecoder.c:3783:gst_video_decoder_negotiate_pool:<omxh264dec-
> > omxh264dec0> Subclass failed to decide allocation
> > 0:00:00.969139166 31143 0x6fd03290 WARN GST_PADS gstpad.c:
> > 4231:gst_pad_peer_query:<omxh264dec-omxh264dec0:src> could not send sticky
> > events
> > 0:00:00.969251770 31143 0x6fd03290 WARN videodecoder
> > gstvideodecoder.c:3783:gst_video_decoder_negotiate_pool:<omxh264dec-
> > omxh264dec0> Subclass failed to decide allocation
> > 0:00:00.972893854 31143 0x6fd03290 WARN GST_PADS gstpad.c:
> > 4231:gst_pad_peer_query:<omxh264dec-omxh264dec0:src> could not send sticky
> > events
> > 0:00:00.988252083 31143 0x6fd03290 WARN omxvideodec
> > gstomxvideodec.c:1927:gst_omx_video_dec_loop:<omxh264dec-omxh264dec0>
> > error: Internal data stream error.
> > 0:00:00.988374999 31143 0x6fd03290 WARN omxvideodec
> > gstomxvideodec.c:1927:gst_omx_video_dec_loop:<omxh264dec-omxh264dec0>
> > error: stream stopped, reason not-negotiated
> > 0:00:00.988662604 31143 0x1fed490 WARN basesrc
> > gstbasesrc.c: 3072:gst_base_src_loop:<rpicamsrc0> error: Internal data
> > stream error. 0:00:00.988741718 31143 0x1fed490 WARN
> > basesrc gstbasesrc.c: 3072:gst_base_src_loop:<rpicamsrc0> error:
> > streaming stopped, reason not- negotiated (-4)
> > ERROR: from element /GstPipeline:pipeline0/GstOMXH264Dec-
> > omxh264dec:omxh264dec-omxh264dec0: Internal data stream error.
> > Additional debug info:
> > ../../gst-omx-1.16.1/omx/gstomxvideodec.c(1927): gst_omx_video_dec_loop
> > (): /
> > GstPipeline:pipeline0/GstOMXH264Dec-omxh264dec:omxh264dec-omxh264dec0:
> > stream stopped, reason not-negotiated
> > Execution ended after 0:00:00.478420208
> > Setting pipeline to PAUSED ...
> > Setting pipeline to READY ...
> > Setting pipeline to NULL ...
> > Freeing pipeline ...
> > ```
> >
> > Do you have an idea of what's going on here?
> >
> > Thanks
> >
> >> On 22/11/19 3:10 am, Milian Wolff wrote:
> >>> Hey all,
> >>>
> >>> on a Raspberry Pi 3b+ gst-inspect-1.0 shows the following src for
> >>> omxh264dec:
> >>>
> >>> ```
> >>>
> >>> video/x-raw(memory:GLMemory)
> >>>
> >>> format: RGBA
> >>>
> >>> width: [ 1, 2147483647 ]
> >>>
> >>> height: [ 1, 2147483647 ]
> >>>
> >>> framerate: [ 0/1, 2147483647/1 ]
> >>>
> >>> ```
> >>>
> >>> Note that this is missing a texture-target specification. What kind of
> >>> memory is this, and how can I leverage it further down? I would like to
> >>> efficiently change video frames using an OpenGL filter. If we take
> >>> gleffects-twirl as an example, it expects the following as sink:
> >>>
> >>>
> >>> ```
> >>>
> >>> video/x-raw(memory:GLMemory):
> >>> format: RGBA
> >>>
> >>> width: [ 1, 2147483647 ]
> >>>
> >>> height: [ 1, 2147483647 ]
> >>>
> >>> framerate: [ 0/1, 2147483647/1 ]
> >>>
> >>> texture-target: 2D
> >>>
> >>> ```
> >>>
> >>> How can I connect the two together? If I try, to add the filter directly
> >>> after the decoder in a pipeline, I get:
> >>>
> >>> ```
> >>> ERROR: from element /GstPipeline:pipeline0/GstOMXH264Dec-
> >>> omxh264dec:omxh264dec-omxh264dec0: Internal data stream error.
> >>> Additional debug info:
> >>> ../../gst-omx-1.16.1/omx/gstomxvideodec.c(1927): gst_omx_video_dec_loop
> >>> (): /
> >>> GstPipeline:pipeline0/GstOMXH264Dec-omxh264dec:omxh264dec-omxh264dec0:
> >>> stream stopped, reason not-negotiated
> >>> ```
> >>>
> >>> Can someone sched some light on this? How can one post-process the
> >>> GLMemory in a gl video filter? Is the missing texture-target a bug in
> >>> omxh264dec, or can I somehow make the raw GLMemory a texture? Or does
> >>> this mean the frame lives in some other form of non-texture memory that
> >>> can be displayed, but not otherwise accessed by OpenGL for video filter
> >>> purposes?
> >>>
> >>> Thanks
> >>>
> >>> PS: In case you are wondering what we are trying to achieve on a higher
> >>> level:
> >>>
> >>> rpicamsrc -> h264 -> omxh264dec -> glmemory -> custom gl filter to add a
> >>> non- static overlay to every frame -> tee to display sink and then also
> >>> a
> >>> h264- encoded avi record sink
> >>>
> >>> _______________________________________________
> >>> gstreamer-devel mailing list
> >>> gstreamer-devel at lists.freedesktop.org
> >>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
--
Milian Wolff | milian.wolff at kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3826 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20191125/594acead/attachment-0001.bin>
More information about the gstreamer-devel
mailing list