What's the meaning of video/x-raw(memory:GLMemory) without texture-target?
Matthew Waters
ystreet00 at gmail.com
Mon Nov 25 22:45:22 UTC 2019
Now you're at my limit if what I can confidently tell you without
experimenting myself :)
I don't know why the twirl effect is not applied unfortunately. I would
need to experiment with that to find the bug there.
The omx decoder GL zerocopy path for the RPi is unfortunately rather
special. In theory you don't need the 'glupload ! glcolorconvert'
however, there may be lifetime/format considerations that glupload !
glcolorconvert will paper over.
On 25/11/19 9:31 pm, Milian Wolff wrote:
> On Monday, November 25, 2019 11:20:00 AM CET Matthew Waters wrote:
>> Try adding a glupload ! glcolorconvert before the gleffects_twirl
>> element. glimagesink contains these automatically already.
> Thanks! This fixes the errors but it leaves me equally puzzled:
>
> Most notably, the twirl effect isn't taking effect, i.e. this is my command
> and debug output now:
>
> ```
> # GST_DEBUG=3 gst-launch-1.0 rpicamsrc ! "video/x-h264,width=640,height=480" !
> h264parse ! omxh264dec ! "video/x-raw(memory:GLMemory)" ! glupload !
> gleffects_twirl ! queue ! glimagesink
> Setting pipeline to PAUSED ...
> Pipeline is live and does not need PREROLL ...
> 0:00:00.489548698 381 0x1c45af0 FIXME default gstutils.c:
> 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random
> stream-id, consider implementing a deterministic way of creating a stream-id
> Got context from element 'sink': gst.gl.GLDisplay=context,
> gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0";
> Setting pipeline to PLAYING ...
> New clock: GstSystemClock
> 0:00:00.880536979 381 0x1c45af0 ERROR omxvideodec
> gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0>
> Empty caps
> 0:00:01.057018906 381 0x1c2f750 FIXME glslstage
> gstglslstage.c:518:_compile_shader:<glslstage0> vertex shader info
> log:Compiled
> 0:00:01.067645364 381 0x1c2f750 FIXME glslstage
> gstglslstage.c:518:_compile_shader:<glslstage1> fragment shader info
> log:Compiled
> 0:00:01.125377187 381 0x1c2f750 FIXME glslstage
> gstglslstage.c:518:_compile_shader:<glslstage2> vertex shader info
> log:Compiled
> 0:00:01.127863541 381 0x1c2f750 FIXME glslstage
> gstglslstage.c:518:_compile_shader:<glslstage3> fragment shader info
> log:Compiled
> 0:00:01.160143698 381 0x1c2f750 FIXME glslstage
> gstglslstage.c:518:_compile_shader:<glslstage4> vertex shader info
> log:Compiled
> 0:00:01.161840260 381 0x1c2f750 FIXME glslstage
> gstglslstage.c:518:_compile_shader:<glslstage5> fragment shader info
> log:Compiled
> ```
>
> But the camera image is displayed as-is. What's odd is that I seem to be
> seeing the twirl effect for a very short time when I "Ctrl+C" terminate the
> `gst-launch` process.
>
> If I use `gltestsrc` as a replacement for the source up to `omxh264dec`, I see
> the twirl effect directly. What's going on here?
>
> Secondly, could you please explain to me why the `glupload ! glcolorconvert`
> is required here? Most notably, `video/x-raw(GLMemory)` should be on the GPU
> already, why do I need to "upload" it again? And the format is already `RGBA`,
> so why do I need glcolorconvert?
>
> Thanks
>
>> On 25/11/19 8:30 pm, Milian Wolff wrote:
>>> On Friday, November 22, 2019 5:12:57 AM CET Matthew Waters wrote:
>>>> Hi,
>>> Hey Matt!
>>>
>>>> By default a 'video/x-raw(memory:GLMemory)' caps without the
>>>> texture-target field is equivalent to texture-target=2D as that was the
>>>> default before we differentiated the different OpenGL texture targets.
>>>> Now elements should be able to negotiate the differences between them
>>>> correctly however there may be bugs.
>>> OK, thanks - good to know!
>>>
>>>> The GLMemory output from omxh264dec is highly dependant on the platform
>>>> you are running on and may or may not work correctly. I know that it
>>>> used to work on the RPi and have not tested any other omx platform
>>>> myself.
>>> What's interesting is that this seems to work fine when I shove the data
>>> directly into the glimagesink, i.e.:
>>>
>>> ```
>>> # GST_DEBUG=3 gst-launch-1.0 rpicamsrc !
>>> "video/x-h264,width=640,height=480" ! h264parse ! omxh264dec !
>>> "video/x-raw(memory:GLMemory)" ! queue !
>>> glimagesinkSetting pipeline to PAUSED ...
>>> Pipeline is live and does not need PREROLL ...
>>> 0:00:00.488648854 31116 0x167e7b0 FIXME default
>>> gstutils.c:
>>> 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random
>>> stream-id, consider implementing a deterministic way of creating a
>>> stream-id Got context from element 'sink': gst.gl.GLDisplay=context,
>>> gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0";
>>> Setting pipeline to PLAYING ...
>>> New clock: GstSystemClock
>>> 0:00:00.894124323 31116 0x167e7b0 ERROR omxvideodec
>>> gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0>
>>> Empty caps
>>> 0:00:00.985837604 31116 0x167e230 FIXME glslstage
>>> gstglslstage.c:518:_compile_shader:<glslstage0> vertex shader info
>>> log:Compiled
>>> 0:00:00.990119375 31116 0x167e230 FIXME glslstage
>>> gstglslstage.c:518:_compile_shader:<glslstage1> fragment shader info
>>> log:Compiled
>>> 0:00:01.013341718 31116 0x167e230 FIXME glslstage
>>> gstglslstage.c:518:_compile_shader:<glslstage2> vertex shader info
>>> log:Compiled
>>> 0:00:01.014671666 31116 0x167e230 FIXME glslstage
>>> gstglslstage.c:518:_compile_shader:<glslstage3> fragment shader info
>>> log:Compiled
>>> ^Chandling interrupt.
>>> Interrupt: Stopping pipeline ...
>>> Execution ended after 0:00:06.801416612
>>> Setting pipeline to PAUSED ...
>>> Setting pipeline to READY ...
>>> Setting pipeline to NULL ...
>>> Freeing pipeline ...
>>> ```
>>>
>>> I don't know what the ERROR is trying to tell me, but this pipeline works
>>> a
>>> charm.
>>>
>>>> The first step to debugging this would be to enable GStreamer debug
>>>> logging to get an idea of what may be wrong.
>>> But the following happens when I add the gleffects_twirl filter (either
>>> before or after the queue, doesn't matter):
>>>
>>> ```
>>> # GST_DEBUG=3 gst-launch-1.0 rpicamsrc !
>>> "video/x-h264,width=640,height=480" ! h264parse ! omxh264dec !
>>> "video/x-raw(memory:GLMemory)" ! gleffects_twirl ! queue ! glimagesink
>>> Setting pipeline to PAUSED ...
>>> Pipeline is live and does not need PREROLL ...
>>> 0:00:00.506736041 31143 0x1fed490 FIXME default
>>> gstutils.c:
>>> 3981:gst_pad_create_stream_id_internal:<rpicamsrc0:src> Creating random
>>> stream-id, consider implementing a deterministic way of creating a
>>> stream-id Got context from element 'sink': gst.gl.GLDisplay=context,
>>> gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayEGL\)\ gldisplayegl0";
>>> Setting pipeline to PLAYING ...
>>> New clock: GstSystemClock
>>> 0:00:00.882512708 31143 0x1fed490 ERROR omxvideodec
>>> gstomxvideodec.c:2079:gst_omx_video_dec_negotiate:<omxh264dec-omxh264dec0>
>>> Empty caps
>>> 0:00:00.968698020 31143 0x6fd03290 WARN videodecoder
>>> gstvideodecoder.c:3783:gst_video_decoder_negotiate_pool:<omxh264dec-
>>> omxh264dec0> Subclass failed to decide allocation
>>> 0:00:00.969139166 31143 0x6fd03290 WARN GST_PADS gstpad.c:
>>> 4231:gst_pad_peer_query:<omxh264dec-omxh264dec0:src> could not send sticky
>>> events
>>> 0:00:00.969251770 31143 0x6fd03290 WARN videodecoder
>>> gstvideodecoder.c:3783:gst_video_decoder_negotiate_pool:<omxh264dec-
>>> omxh264dec0> Subclass failed to decide allocation
>>> 0:00:00.972893854 31143 0x6fd03290 WARN GST_PADS gstpad.c:
>>> 4231:gst_pad_peer_query:<omxh264dec-omxh264dec0:src> could not send sticky
>>> events
>>> 0:00:00.988252083 31143 0x6fd03290 WARN omxvideodec
>>> gstomxvideodec.c:1927:gst_omx_video_dec_loop:<omxh264dec-omxh264dec0>
>>> error: Internal data stream error.
>>> 0:00:00.988374999 31143 0x6fd03290 WARN omxvideodec
>>> gstomxvideodec.c:1927:gst_omx_video_dec_loop:<omxh264dec-omxh264dec0>
>>> error: stream stopped, reason not-negotiated
>>> 0:00:00.988662604 31143 0x1fed490 WARN basesrc
>>> gstbasesrc.c: 3072:gst_base_src_loop:<rpicamsrc0> error: Internal data
>>> stream error. 0:00:00.988741718 31143 0x1fed490 WARN
>>> basesrc gstbasesrc.c: 3072:gst_base_src_loop:<rpicamsrc0> error:
>>> streaming stopped, reason not- negotiated (-4)
>>> ERROR: from element /GstPipeline:pipeline0/GstOMXH264Dec-
>>> omxh264dec:omxh264dec-omxh264dec0: Internal data stream error.
>>> Additional debug info:
>>> ../../gst-omx-1.16.1/omx/gstomxvideodec.c(1927): gst_omx_video_dec_loop
>>> (): /
>>> GstPipeline:pipeline0/GstOMXH264Dec-omxh264dec:omxh264dec-omxh264dec0:
>>> stream stopped, reason not-negotiated
>>> Execution ended after 0:00:00.478420208
>>> Setting pipeline to PAUSED ...
>>> Setting pipeline to READY ...
>>> Setting pipeline to NULL ...
>>> Freeing pipeline ...
>>> ```
>>>
>>> Do you have an idea of what's going on here?
>>>
>>> Thanks
>>>
>>>> On 22/11/19 3:10 am, Milian Wolff wrote:
>>>>> Hey all,
>>>>>
>>>>> on a Raspberry Pi 3b+ gst-inspect-1.0 shows the following src for
>>>>> omxh264dec:
>>>>>
>>>>> ```
>>>>>
>>>>> video/x-raw(memory:GLMemory)
>>>>>
>>>>> format: RGBA
>>>>>
>>>>> width: [ 1, 2147483647 ]
>>>>>
>>>>> height: [ 1, 2147483647 ]
>>>>>
>>>>> framerate: [ 0/1, 2147483647/1 ]
>>>>>
>>>>> ```
>>>>>
>>>>> Note that this is missing a texture-target specification. What kind of
>>>>> memory is this, and how can I leverage it further down? I would like to
>>>>> efficiently change video frames using an OpenGL filter. If we take
>>>>> gleffects-twirl as an example, it expects the following as sink:
>>>>>
>>>>>
>>>>> ```
>>>>>
>>>>> video/x-raw(memory:GLMemory):
>>>>> format: RGBA
>>>>>
>>>>> width: [ 1, 2147483647 ]
>>>>>
>>>>> height: [ 1, 2147483647 ]
>>>>>
>>>>> framerate: [ 0/1, 2147483647/1 ]
>>>>>
>>>>> texture-target: 2D
>>>>>
>>>>> ```
>>>>>
>>>>> How can I connect the two together? If I try, to add the filter directly
>>>>> after the decoder in a pipeline, I get:
>>>>>
>>>>> ```
>>>>> ERROR: from element /GstPipeline:pipeline0/GstOMXH264Dec-
>>>>> omxh264dec:omxh264dec-omxh264dec0: Internal data stream error.
>>>>> Additional debug info:
>>>>> ../../gst-omx-1.16.1/omx/gstomxvideodec.c(1927): gst_omx_video_dec_loop
>>>>> (): /
>>>>> GstPipeline:pipeline0/GstOMXH264Dec-omxh264dec:omxh264dec-omxh264dec0:
>>>>> stream stopped, reason not-negotiated
>>>>> ```
>>>>>
>>>>> Can someone sched some light on this? How can one post-process the
>>>>> GLMemory in a gl video filter? Is the missing texture-target a bug in
>>>>> omxh264dec, or can I somehow make the raw GLMemory a texture? Or does
>>>>> this mean the frame lives in some other form of non-texture memory that
>>>>> can be displayed, but not otherwise accessed by OpenGL for video filter
>>>>> purposes?
>>>>>
>>>>> Thanks
>>>>>
>>>>> PS: In case you are wondering what we are trying to achieve on a higher
>>>>> level:
>>>>>
>>>>> rpicamsrc -> h264 -> omxh264dec -> glmemory -> custom gl filter to add a
>>>>> non- static overlay to every frame -> tee to display sink and then also
>>>>> a
>>>>> h264- encoded avi record sink
>>>>>
>>>>> _______________________________________________
>>>>> gstreamer-devel mailing list
>>>>> gstreamer-devel at lists.freedesktop.org
>>>>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20191126/1653fdf2/attachment.sig>
More information about the gstreamer-devel
mailing list