Readback OpenGL texture from omxh264dec
Matthew Waters
ystreet00 at gmail.com
Wed Jun 24 04:27:27 UTC 2020
Hi,
So, GStreamer also attempts to g_module_open() and g_module_symbol() all
OpenGL functions from a specific library as not all OpenGL functions are
exported from all OpenGL libraries. That library name is by default,
libGLES2v2 as outlined here:
https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/blob/master/gst-libs/gst/gl/gstglcontext.c#L111
but can be configured with the meson build option: 'gles2_module_name'.
The same thing occurs for libEGL and can be overridden by the meson
build option 'egl_module_name'.
Hope the helps
Cheers
-Matt
On 24/6/20 1:12 pm, Rafael Savignon wrote:
> Thanks Matthew,
> to confirm it i did a little patch to check if the passed GstMemory
> was indeed a GstGLMemoryEGL, but apparently it is not, the below
> assert failed right after the qt video sink received a new frame.
>
> GstMemory *mem = gst_buffer_peek_memory (buffer, 0);
> g_assert (gst_is_gl_memory_egl (mem));
>
> As you suggested, I tried to execute the pipeline 'omxh264dec !
> glimagesinkelement', but it ends up giving me an error when calling
> glGetString(GL_VERSION).
>
> $ gst-launch-1.0 filesrc location=/home/root/test.mp4 ! omxh264dec !
> glimagesinkelement
>
> glcontext
> gstglcontext_egl.c:464:gst_gl_context_egl_choose_config:<glcontextegl0>
> chosen EGLConfig
>
> glcontext
> gstglcontext_egl.c:174:gst_gl_context_egl_dump_config:<glcontextegl0>
> dumping EGLConfig 0x4 with id 0x4 and native visual id 0x8428 of type
> 0x3038
> glcontext
> gstglcontext_egl.c:207:gst_gl_context_egl_dump_config:<glcontextegl0>
> Conformant for OpenGL ES|OpenGL ES 2.x|OpenVG
>
> glcontext
> gstglcontext_egl.c:240:gst_gl_context_egl_dump_config:<glcontextegl0>
> Renderable for OpenGL ES|OpenGL ES 2.x|OpenVG
>
> glcontext
> gstglcontext_egl.c:273:gst_gl_context_egl_dump_config:<glcontextegl0>
> Surface for
> window|pbuffer|multisample-resolve-box|swap-behaviour-preserved|vg-alpha-format-pre|vg-colorspace-linear
> glcontext
> gstglcontext_egl.c:313:gst_gl_context_egl_dump_config:<glcontextegl0>
> [R, G, B, A] = [8, 8, 8, 0]
>
> glcontext
> gstglcontext_egl.c:335:gst_gl_context_egl_dump_config:<glcontextegl0>
> [D, S] = [24, 0]
>
> glcontext
> gstglcontext_egl.c:346:gst_gl_context_egl_dump_config:<glcontextegl0>
> Swap interval range is [0, 2147483647]
>
> glcontext
> gstglcontext_egl.c:360:gst_gl_context_egl_dump_config:<glcontextegl0>
> PBuffer maximum dimensions are [2048, 2048]. Max pixels are 4194304
>
> glcontext
> gstglcontext_egl.c:373:gst_gl_context_egl_dump_config:<glcontextegl0>
> Multisample buffers: 0 and Samples per pixel: 0
>
> glcontext
> gstglcontext_egl.c:501:_create_context_with_flags:<glcontextegl0>
> attempting to create OpenGL ES context version 2.0 flags 0 profile 0
>
> glcontext gstglcontext_egl.c:746:gst_gl_context_egl_create_context:
> gl context created: 1
>
> glwindow gstglwindow_dispmanx_egl.c:226:window_resize: resizing
> invisible window from 0x0 to 16x16
>
> videosink gstvideosink.c:132:gst_video_sink_center_rect: source is
> 16x16 dest is 640x480, result is 16x16 with x,y 312x232
>
> glcontext gstglcontext_egl.c:794:gst_gl_context_egl_create_context:
> Creating EGLSurface from window_handle 0x5fd100
>
> glcontext gstglcontext_egl.c:828:gst_gl_context_egl_create_context:
> surface created
>
> glcontext
> gstglcontext.c:1247:gst_gl_context_create_thread:<glcontextegl0>
> created context
>
> glcontext gstglcontext.c:750:gst_gl_context_activate:<glcontextegl0>
> activate:1
>
> glcontext
> gstglcontext.c:1263:gst_gl_context_create_thread:<glcontextegl0>
> available GL APIs: gles2
>
> glcontext
> gstglcontext.c:1282:gst_gl_context_create_thread:<glcontextegl0>
> Filling info
>
> glcontext gstglcontext.c:1042:gst_gl_context_create:<glcontextegl0>
> gl thread created
>
> glimagesink gstglimagesink.c:1010:_ensure_gl_setup:<glimagesink0>
> *error: glGetString not defined or returned invalid value*
>
> After some searching I find out that this problem could be related
> with the wrong GL lib loading. So I certified that gst-launch-1.0 was
> loading the correct glGetString version.
>
> My system listed two libraries with the symbol:
>
> $ find . -name \*.so -exec bash -c "nm --defined-only -D {}
> 2>/dev/null | grep glGetString && echo {}" \;
> 00008810 T glGetString
> ./*libbrcmGLESv2.so*
> 0000880c T glGetString
> ./*libGLESv2.so*
>
> According to rpi doc, the corrected one is the brcm. Thus, I ran gdb
> to check which one is being loaded right before
> glGetString(GL_VERSION) was called. gst_gl_display_create_context was
> chosen to the the point of verification.
>
> $ gdb gst-launch-1.0
> Thread 1 "gst-launch-1.0" hit Breakpoint 2, 0x7693fa22 in
> gst_gl_display_create_context () from /usr/lib/libgstgl-1.0.so.0
> (gdb) info sharedlibrary
> From To Syms Read Shared Object Library
> 0x76fd5b80 0x76fe96b0 Yes (*) /lib/ld-linux-armhf.so.3
> 0x76f0d9d8 0x76f94684 Yes (*) /usr/lib/libgstreamer-1.0.so.0
> 0x76e1d7e8 0x76e7850c Yes (*) /usr/lib/libglib-2.0.so.0
> 0x76dc2c40 0x76de9810 Yes (*) /usr/lib/libgobject-2.0.so.0
> 0x76d99540 0x76da4b94 Yes (*) /lib/libpthread.so.0
> 0x76cbba40 0x76d61ac4 Yes (*) /lib/libc.so.6
> 0x76c8dbb0 0x76c8e6dc Yes (*) /usr/lib/libgmodule-2.0.so.0
> 0x76c32148 0x76c59c60 Yes (*) /lib/libm.so.6
> 0x76c16b38 0x76c17518 Yes (*) /lib/libdl.so.2
> 0x76bb8f48 0x76befd8c Yes (*) /usr/lib/libpcre.so.1
> 0x76ba3360 0x76ba6680 Yes (*) /usr/lib/libffi.so.7
> 0x76b85118 0x76b8ff10 Yes (*) /lib/libgcc_s.so.1
> 0x76b28520 0x76b59774 Yes (*)
> /usr/lib/gstreamer-1.0/libgstcoreelements.so
> 0x76ac55e8 0x76b012a4 Yes (*) /usr/lib/libgstbase-1.0.so.0
> 0x76a77e08 0x76a9ce4c Yes (*) /usr/lib/gstreamer-1.0/libgstomx.so
> 0x769fedd0 0x76a4623c Yes (*) /usr/lib/libgstvideo-1.0.so.0
> 0x76998058 0x769ce23c Yes (*) /usr/lib/libgstaudio-1.0.so.0
> 0x7697ad60 0x7697b910 Yes (*) /usr/lib/libgstallocators-1.0.so.0
> 0x769355a0 0x76957ae8 Yes (*) /usr/lib/libgstgl-1.0.so.0
> 0x768ccfe8 0x7690893c Yes (*) /usr/lib/liborc-0.4.so.0
> 0x76892618 0x768a9e14 Yes (*) /usr/lib/libgsttag-1.0.so.0
> 0x76878a0c 0x76879ed4 Yes (*) /usr/lib/libbcm_host.so
> 0x7685114c 0x768642dc Yes (*) */usr/lib/libbrcmEGL.so*
> 0x76829928 0x76831dc4 Yes (*) /usr/lib/libvchostif.so
> 0x76806dc8 0x76812f2c Yes (*) */usr/lib/libbrcmGLESv2.so*
> 0x767e3bc8 0x767ecc88 Yes (*) /lib/libz.so.1
> 0x767cd704 0x767cfb30 Yes (*) /usr/lib/libvchiq_arm.so
> 0x767b6c4c 0x767b9e28 Yes (*) /usr/lib/libvcos.so
> 0x7679fae0 0x767a1f30 Yes (*) /lib/librt.so.1
> 0x76764228 0x7677d648 Yes (*) /usr/lib/gstreamer-1.0/libgstopengl.so
> 0x76742240 0x76748fb4 Yes (*) /usr/lib/libgstcontroller-1.0.so.0
> 0x767135b0 0x7672a9f0 Yes (*) /usr/lib/libpng16.so.16
> 0x766d89e0 0x766fb670 Yes (*) /usr/lib/libjpeg.so.62
>
> It's picking the right lib (libbrcmGLESv2.so) so glGetString is
> returning a invalid value.
> Finally, to validate i coded a simple sample to call glGetString,
> which surprisingly returned : *OpenGL ES 2.0*
>
> What could be wrong with glimagesinkelement ?
>
> regards.
>
>
> On Sun, Jun 21, 2020 at 11:39 AM Matthew Waters <ystreet00 at gmail.com
> <mailto:ystreet00 at gmail.com>> wrote:
>
> The OpenGL texture out of omxh264dec on the RPi is wrapped in a
> GstGLMemoryEGL which already wraps the EGLImage produced by the
> OMX decoder. I'm not entirely sure what VCSM is or how it
> actually relates to all this but I assume that the following
> pipeline works for you? 'omxh264dec ! glimagesinkelement' If so,
> that is already using the EGLImage/OpenGL texture produced by
> omxh264dec and rendering using OpenGL with glimagesink. It is
> very hard in general for the RPi to do anything but only decode
> and display a 1080p at 30 video and even then it can just barely do it.
>
> On 21/6/20 10:22 pm, Rafael Savignon wrote:
>> Hi all,
>> I'm developing a Qt5 application to playback hardware decoded
>> h.264 files. But on raspberry pi 3 its struggling to play media
>> greater than 720p. After some investigation
>> of *QGstVideoBuffer* class I realized that calls
>> to gst_video_frame_map function passing bigger frames were
>> consuming much cpu time ~50ms, which was resulting in the
>> postponing of next frames delivery. So I had to find out a more
>> performatic way to read the video frame content. After some
>> search I found out that a special buffer mechanism could enable
>> applications to map GPU memory directly to addressable process
>> memory, avoiding unnecessary copy (VCSM). Then I took another
>> look at QtMultimedia source code and saw that a OpenGL texture
>> handle was being packaged in with *QGstVideoBuffer*, as shown below.
>> guint *textureId* = gst_gl_memory_get_texture_id(glmem);
>> videoBuffer = new QGstVideoBuffer(buffer, m_videoInfo,
>> m_format.handleType(), *textureId*); *
>> *
>>
>> So I thought that things were starting to get clear. Thus I
>> decided to use vcsm to map the texture and speed things up .
>>
>> First, on my *QAbstractVideoSurface::start* implemented method I
>> initialized vcsm and created a EGLimageKHR.
>>
>> int w = Util::nextPOT(size.width());
>> int h = Util::nextPOT(size.height());
>> const EGLint attrib[] = { EGL_IMAGE_PRESERVED_KHR, EGL_TRUE,
>> EGL_NONE, EGL_NONE };
>> vcsm_info.width = w;
>> vcsm_info.height = h;
>>
>> vcsm_init();
>> eglFbImage = eglCreateImageKHR(eglGetCurrentDisplay(),
>> EGL_NO_CONTEXT, EGL_IMAGE_BRCM_VCSM, &vcsm_info, attrib);
>>
>> Second, on my *QAbstractVideoSurface::present* implementation
>> method I tried to grab the passed Texture content, but it didn't
>> work. Despite this, all OpenGL calls succeeded. I can access the
>> mapped buffer, but it doesn't contain anything meaningful.
>>
>> QOpenGLFunctions* f = ctx->functions();
>> GLuint framebuffer;
>> GLuint depthRenderbuffer;
>> GLint prevFbo;
>> GLenum status = GL_FRAMEBUFFER_COMPLETE;
>> GLuint texture = static_cast<GLuint>(
>> currentFrame.handle().toInt() );
>> int texWidth = Util::nextPOT(currentFrame.width());
>> int texHeight = Util::nextPOT(currentFrame.height());
>>
>> GLCHK(f->glGetIntegerv( GL_FRAMEBUFFER_BINDING, &prevFbo ));
>> GLCHK(f->glGenFramebuffers(1, &framebuffer));
>> GLCHK(f->glBindFramebuffer(GL_FRAMEBUFFER, framebuffer));
>> GLCHK(glActiveTexture(GL_TEXTURE0));
>> GLCHK(glBindTexture(GL_TEXTURE_2D, texture));
>> GLCHK(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER,
>> GL_NEAREST));
>> GLCHK(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER,
>> GL_NEAREST));
>> GLCHK(glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, eglFbImage));
>> GLCHK(f->glFramebufferTexture2D(GL_FRAMEBUFFER,
>> GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, texture, 0));
>> GLCHK(glBindTexture(GL_TEXTURE_2D, 0));
>> status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
>> // check fbo status
>> GLCHK(f->glFinish());
>>
>> uint8_t *vcsmBuffer;
>> VCSM_CACHE_TYPE_T cacheType;
>> vcsmBuffer = (uint8_t*)vcsm_lock_cache(vcsm_info.vcsm_handle,
>> VCSM_CACHE_TYPE_HOST, &cacheType);
>> // print buffer
>> vcsm_unlock_ptr(vcsmBuffer);
>>
>> Is the Texture Id correctly filled by omxh264dec or glupload ?
>> Can i bind the texture id to a different fbo ?
>> Why *gst_video_frame_map *correctly map the video frame content
>> and vcsm do not despite being addressable?
>> *
>> *
>> If someone could give a tip on this I would be thankful.
>>
>> regards.
>>
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org <mailto:gstreamer-devel at lists.freedesktop.org>
>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20200624/481b0abc/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20200624/481b0abc/attachment-0001.sig>
More information about the gstreamer-devel
mailing list