Readback OpenGL texture from omxh264dec
Rafael Savignon
rsavignon at gmail.com
Wed Jun 24 03:12:20 UTC 2020
Thanks Matthew,
to confirm it i did a little patch to check if the passed GstMemory was
indeed a GstGLMemoryEGL, but apparently it is not, the below assert failed
right after the qt video sink received a new frame.
GstMemory *mem = gst_buffer_peek_memory (buffer, 0);
g_assert (gst_is_gl_memory_egl (mem));
As you suggested, I tried to execute the pipeline 'omxh264dec !
glimagesinkelement', but it ends up giving me an error when calling
glGetString(GL_VERSION).
$ gst-launch-1.0 filesrc location=/home/root/test.mp4 ! omxh264dec !
glimagesinkelement
glcontext
gstglcontext_egl.c:464:gst_gl_context_egl_choose_config:<glcontextegl0>
chosen EGLConfig
glcontext
gstglcontext_egl.c:174:gst_gl_context_egl_dump_config:<glcontextegl0>
dumping EGLConfig 0x4 with id 0x4 and native visual id 0x8428 of type
0x3038
glcontext
gstglcontext_egl.c:207:gst_gl_context_egl_dump_config:<glcontextegl0>
Conformant for OpenGL ES|OpenGL ES 2.x|OpenVG
glcontext
gstglcontext_egl.c:240:gst_gl_context_egl_dump_config:<glcontextegl0>
Renderable for OpenGL ES|OpenGL ES 2.x|OpenVG
glcontext
gstglcontext_egl.c:273:gst_gl_context_egl_dump_config:<glcontextegl0>
Surface for
window|pbuffer|multisample-resolve-box|swap-behaviour-preserved|vg-alpha-format-pre|vg-colorspace-linear
glcontext
gstglcontext_egl.c:313:gst_gl_context_egl_dump_config:<glcontextegl0> [R,
G, B, A] = [8, 8, 8, 0]
glcontext
gstglcontext_egl.c:335:gst_gl_context_egl_dump_config:<glcontextegl0> [D,
S] = [24, 0]
glcontext
gstglcontext_egl.c:346:gst_gl_context_egl_dump_config:<glcontextegl0> Swap
interval range is [0, 2147483647]
glcontext
gstglcontext_egl.c:360:gst_gl_context_egl_dump_config:<glcontextegl0>
PBuffer maximum dimensions are [2048, 2048]. Max pixels are 4194304
glcontext
gstglcontext_egl.c:373:gst_gl_context_egl_dump_config:<glcontextegl0>
Multisample buffers: 0 and Samples per pixel: 0
glcontext
gstglcontext_egl.c:501:_create_context_with_flags:<glcontextegl0>
attempting to create OpenGL ES context version 2.0 flags 0 profile 0
glcontext gstglcontext_egl.c:746:gst_gl_context_egl_create_context: gl
context created: 1
glwindow gstglwindow_dispmanx_egl.c:226:window_resize: resizing invisible
window from 0x0 to 16x16
videosink gstvideosink.c:132:gst_video_sink_center_rect: source is 16x16
dest is 640x480, result is 16x16 with x,y 312x232
glcontext gstglcontext_egl.c:794:gst_gl_context_egl_create_context:
Creating EGLSurface from window_handle 0x5fd100
glcontext gstglcontext_egl.c:828:gst_gl_context_egl_create_context:
surface created
glcontext
gstglcontext.c:1247:gst_gl_context_create_thread:<glcontextegl0> created
context
glcontext gstglcontext.c:750:gst_gl_context_activate:<glcontextegl0>
activate:1
glcontext
gstglcontext.c:1263:gst_gl_context_create_thread:<glcontextegl0> available
GL APIs: gles2
glcontext
gstglcontext.c:1282:gst_gl_context_create_thread:<glcontextegl0> Filling
info
glcontext gstglcontext.c:1042:gst_gl_context_create:<glcontextegl0> gl
thread created
glimagesink gstglimagesink.c:1010:_ensure_gl_setup:<glimagesink0> *error:
glGetString not defined or returned invalid value*
After some searching I find out that this problem could be related with the
wrong GL lib loading. So I certified that gst-launch-1.0 was loading the
correct glGetString version.
My system listed two libraries with the symbol:
$ find . -name \*.so -exec bash -c "nm --defined-only -D {} 2>/dev/null |
grep glGetString && echo {}" \;
00008810 T glGetString
./*libbrcmGLESv2.so*
0000880c T glGetString
./*libGLESv2.so*
According to rpi doc, the corrected one is the brcm. Thus, I ran gdb to
check which one is being loaded right before glGetString(GL_VERSION) was
called. gst_gl_display_create_context was chosen to the the point of
verification.
$ gdb gst-launch-1.0
Thread 1 "gst-launch-1.0" hit Breakpoint 2, 0x7693fa22 in
gst_gl_display_create_context () from /usr/lib/libgstgl-1.0.so.0
(gdb) info sharedlibrary
>From To Syms Read Shared Object Library
0x76fd5b80 0x76fe96b0 Yes (*) /lib/ld-linux-armhf.so.3
0x76f0d9d8 0x76f94684 Yes (*) /usr/lib/libgstreamer-1.0.so.0
0x76e1d7e8 0x76e7850c Yes (*) /usr/lib/libglib-2.0.so.0
0x76dc2c40 0x76de9810 Yes (*) /usr/lib/libgobject-2.0.so.0
0x76d99540 0x76da4b94 Yes (*) /lib/libpthread.so.0
0x76cbba40 0x76d61ac4 Yes (*) /lib/libc.so.6
0x76c8dbb0 0x76c8e6dc Yes (*) /usr/lib/libgmodule-2.0.so.0
0x76c32148 0x76c59c60 Yes (*) /lib/libm.so.6
0x76c16b38 0x76c17518 Yes (*) /lib/libdl.so.2
0x76bb8f48 0x76befd8c Yes (*) /usr/lib/libpcre.so.1
0x76ba3360 0x76ba6680 Yes (*) /usr/lib/libffi.so.7
0x76b85118 0x76b8ff10 Yes (*) /lib/libgcc_s.so.1
0x76b28520 0x76b59774 Yes (*)
/usr/lib/gstreamer-1.0/libgstcoreelements.so
0x76ac55e8 0x76b012a4 Yes (*) /usr/lib/libgstbase-1.0.so.0
0x76a77e08 0x76a9ce4c Yes (*) /usr/lib/gstreamer-1.0/libgstomx.so
0x769fedd0 0x76a4623c Yes (*) /usr/lib/libgstvideo-1.0.so.0
0x76998058 0x769ce23c Yes (*) /usr/lib/libgstaudio-1.0.so.0
0x7697ad60 0x7697b910 Yes (*) /usr/lib/libgstallocators-1.0.so.0
0x769355a0 0x76957ae8 Yes (*) /usr/lib/libgstgl-1.0.so.0
0x768ccfe8 0x7690893c Yes (*) /usr/lib/liborc-0.4.so.0
0x76892618 0x768a9e14 Yes (*) /usr/lib/libgsttag-1.0.so.0
0x76878a0c 0x76879ed4 Yes (*) /usr/lib/libbcm_host.so
0x7685114c 0x768642dc Yes (*) */usr/lib/libbrcmEGL.so*
0x76829928 0x76831dc4 Yes (*) /usr/lib/libvchostif.so
0x76806dc8 0x76812f2c Yes (*) */usr/lib/libbrcmGLESv2.so*
0x767e3bc8 0x767ecc88 Yes (*) /lib/libz.so.1
0x767cd704 0x767cfb30 Yes (*) /usr/lib/libvchiq_arm.so
0x767b6c4c 0x767b9e28 Yes (*) /usr/lib/libvcos.so
0x7679fae0 0x767a1f30 Yes (*) /lib/librt.so.1
0x76764228 0x7677d648 Yes (*) /usr/lib/gstreamer-1.0/libgstopengl.so
0x76742240 0x76748fb4 Yes (*) /usr/lib/libgstcontroller-1.0.so.0
0x767135b0 0x7672a9f0 Yes (*) /usr/lib/libpng16.so.16
0x766d89e0 0x766fb670 Yes (*) /usr/lib/libjpeg.so.62
It's picking the right lib (libbrcmGLESv2.so) so glGetString is returning a
invalid value.
Finally, to validate i coded a simple sample to call glGetString, which
surprisingly returned : *OpenGL ES 2.0*
What could be wrong with glimagesinkelement ?
regards.
On Sun, Jun 21, 2020 at 11:39 AM Matthew Waters <ystreet00 at gmail.com> wrote:
> The OpenGL texture out of omxh264dec on the RPi is wrapped in a
> GstGLMemoryEGL which already wraps the EGLImage produced by the OMX
> decoder. I'm not entirely sure what VCSM is or how it actually relates to
> all this but I assume that the following pipeline works for you?
> 'omxh264dec ! glimagesinkelement' If so, that is already using the
> EGLImage/OpenGL texture produced by omxh264dec and rendering using OpenGL
> with glimagesink. It is very hard in general for the RPi to do anything
> but only decode and display a 1080p at 30 video and even then it can just
> barely do it.
>
> On 21/6/20 10:22 pm, Rafael Savignon wrote:
>
> Hi all,
> I'm developing a Qt5 application to playback hardware decoded h.264 files.
> But on raspberry pi 3 its struggling to play media greater than 720p. After
> some investigation of *QGstVideoBuffer* class I realized that calls
> to gst_video_frame_map function passing bigger frames were consuming much
> cpu time ~50ms, which was resulting in the postponing of next frames
> delivery. So I had to find out a more performatic way to read the video
> frame content. After some search I found out that a special buffer
> mechanism could enable applications to map GPU memory directly to
> addressable process memory, avoiding unnecessary copy (VCSM). Then I took
> another look at QtMultimedia source code and saw that a OpenGL texture
> handle was being packaged in with *QGstVideoBuffer*, as shown below.
> guint *textureId* = gst_gl_memory_get_texture_id(glmem);
> videoBuffer = new QGstVideoBuffer(buffer, m_videoInfo,
> m_format.handleType(), *textureId*);
>
> So I thought that things were starting to get clear. Thus I decided to use
> vcsm to map the texture and speed things up .
>
> First, on my *QAbstractVideoSurface::start* implemented method I
> initialized vcsm and created a EGLimageKHR.
>
> int w = Util::nextPOT(size.width());
> int h = Util::nextPOT(size.height());
> const EGLint attrib[] = { EGL_IMAGE_PRESERVED_KHR, EGL_TRUE, EGL_NONE,
> EGL_NONE };
> vcsm_info.width = w;
> vcsm_info.height = h;
>
> vcsm_init();
> eglFbImage = eglCreateImageKHR(eglGetCurrentDisplay(), EGL_NO_CONTEXT,
> EGL_IMAGE_BRCM_VCSM, &vcsm_info, attrib);
>
> Second, on my *QAbstractVideoSurface::present* implementation method I
> tried to grab the passed Texture content, but it didn't work. Despite this,
> all OpenGL calls succeeded. I can access the mapped buffer, but it doesn't
> contain anything meaningful.
>
> QOpenGLFunctions* f = ctx->functions();
> GLuint framebuffer;
> GLuint depthRenderbuffer;
> GLint prevFbo;
> GLenum status = GL_FRAMEBUFFER_COMPLETE;
> GLuint texture = static_cast<GLuint>( currentFrame.handle().toInt() );
> int texWidth = Util::nextPOT(currentFrame.width());
> int texHeight = Util::nextPOT(currentFrame.height());
>
> GLCHK(f->glGetIntegerv( GL_FRAMEBUFFER_BINDING, &prevFbo ));
> GLCHK(f->glGenFramebuffers(1, &framebuffer));
> GLCHK(f->glBindFramebuffer(GL_FRAMEBUFFER, framebuffer));
> GLCHK(glActiveTexture(GL_TEXTURE0));
> GLCHK(glBindTexture(GL_TEXTURE_2D, texture));
> GLCHK(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST));
> GLCHK(glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST));
> GLCHK(glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, eglFbImage));
> GLCHK(f->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
> GL_TEXTURE_2D, texture, 0));
> GLCHK(glBindTexture(GL_TEXTURE_2D, 0));
> status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
> // check fbo status
> GLCHK(f->glFinish());
>
> uint8_t *vcsmBuffer;
> VCSM_CACHE_TYPE_T cacheType;
> vcsmBuffer = (uint8_t*)vcsm_lock_cache(vcsm_info.vcsm_handle,
> VCSM_CACHE_TYPE_HOST, &cacheType);
> // print buffer
> vcsm_unlock_ptr(vcsmBuffer);
>
> Is the Texture Id correctly filled by omxh264dec or glupload ?
> Can i bind the texture id to a different fbo ?
> Why *gst_video_frame_map *correctly map the video frame content and vcsm
> do not despite being addressable?
>
> If someone could give a tip on this I would be thankful.
>
> regards.
>
>
> _______________________________________________
> gstreamer-devel mailing listgstreamer-devel at lists.freedesktop.orghttps://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20200624/cf86de28/attachment-0001.htm>
More information about the gstreamer-devel
mailing list