push opengl texture to gstreamer pipeline
Matthew Waters
ystreet00 at gmail.com
Wed Aug 18 03:42:48 UTC 2021
Hi,
I think I'm at the end if what I can debug over email. In theory
everything should work and GStreamer has examples and elements that do
this exact process of sharing the OpenGL context between GStreamer and
the application/other GUI toolkits.
Please look through the debug logs and any existing
applications/elements (qmlglsink, gtkglsink) that perform this
transformation from the application provided OpenGL context (e.g. from
Qt or Gtk).
On 18/8/21 12:13 am, Lusine Hayrapetyan wrote:
> WRT /" retrieve the GStreamer created OpenGL context from one of the
> downstream elements" /unfortunately it's not our use case.
If it's not your case, you can also perform the necessary GST_CONTEXT
QUERY to retrieve the downstream OpenGL context from GStreamer.
> WRT " /If you are not also providing your wrapped OpenGL in response
> to the 'need-context' message" /I'm providing it, this is how I handle
> "need-context":
> static void sync_bus_call (GstBus * bus, GstMessage * msg, gpointer *
> data)
> {
> APP_STATE_T *state = (APP_STATE_T *)data;
> switch (GST_MESSAGE_TYPE (msg))
> {
> case GST_MESSAGE_NEED_CONTEXT:
> {
> g_print("-------------------------sync_bus_call got
> GST_MESSAGE_NEED_CONTEXT--------------------\n");
>
> const gchar *context_type;
> gst_message_parse_context_type (msg, &context_type);
>
> GstContext *context = NULL;
> if (g_strcmp0 (context_type, GST_GL_DISPLAY_CONTEXT_TYPE)
> == 0)
> {
> g_print("-------------------------got
> GST_GL_DISPLAY_CONTEXT_TYPE--------------------\n");
> context = gst_context_new (GST_GL_DISPLAY_CONTEXT_TYPE,
> TRUE);
>
> // state->x_gst_display was created like
> this: state->x_gst_display =
> gst_gl_display_x11_new_with_display(state->xdisplay);
> gst_context_set_gl_display(context,
> GST_GL_DISPLAY(state->x_gst_display));
>
> g_print("msg->src element is %s\n\n",
> GST_ELEMENT_NAME(msg->src)); // This prints gldownload.
> gst_element_set_context (GST_ELEMENT(msg->src), context);
> }
> else if (g_strcmp0 (context_type, "gst.gl.app_context") == 0)
> {
> g_print("-------------------------got
> gst.gl.app_context--------------------\n");
>
> GstContext *context =
> gst_context_new("gst.gl.app_context", TRUE);
> GstStructure *s = gst_context_writable_structure
> (context);
>
> // state->gl_context is my wrapped context which was
> created like this:
> // state->gl_context = gst_gl_context_new_wrapped
> (GST_GL_DISPLAY (state->x_gst_display),
> // (guintptr) state->context, GST_GL_PLATFORM_EGL,
> GST_GL_API_GLES2);
> gst_structure_set (s, "context", GST_TYPE_GL_CONTEXT,
> state->gl_context, NULL);
>
> g_print("msg->src element is %s\n",
> GST_ELEMENT_NAME(msg->src)); // This prints gldownload.
> gst_element_set_context(GST_ELEMENT(msg->src), context);
> }
> break;
> }
This looks correct from a cursory glance.
> WRT: /"if you are creating another unwrapped OpenGL context with
> gst_gl_display_create_context() then you have notify GStreamer about
> this somehow"/
> I use the unwrapped gl context with /gst_gl_display_add_context /and
> when I'm creating GstGLMemory, is there other places I need to use it?/
> /
No. That should be sufficient.
Cheers
-Matt
> Best Regards,
> Lusine
>
>
> On Tue, Aug 17, 2021 at 4:52 PM Matthew Waters <ystreet00 at gmail.com
> <mailto:ystreet00 at gmail.com>> wrote:
>
> Ah,
>
> Ok. Now, if you are creating another unwrapped OpenGL context
> with gst_gl_display_create_context() then you have notify
> GStreamer about this somehow, or GStreamer will also create it's
> own. If you are not also providing your wrapped OpenGL in
> response to the 'need-context' message, then GStreamer OpenGL is
> not linked with your application OpenGL context.
>
> You should probably just retrieve the GStreamer created OpenGL
> context from one of the downstream elements (the 'context'
> property on most OpenGL elements) and use that.
>
> Cheers
> -Matt
>
> On 17/8/21 1:53 am, Lusine Hayrapetyan wrote:
>> Got it! Thanks!
>> Used /gst_gl_display_x11_new_with_display /instead of
>> /gst_gl_display_egl_new_with_egl_display /and the warning about "
>> /Wrong display type" /is fixed. I don't see any other warn/issue
>> in the gl log but my initial issue still exists 😿 - I create
>> GstGLMemory from texture_id and push the GstBuffer to appsrc and
>> see this errors:
>>
>> 0:00:14.023472773 306156 0x55a56659d6a0 LOG glbasememory
>> gstglbasememory.c:269:_map_data_gl: mapping mem 0x7f105c227d00
>> flags 20002
>> 0:00:14.023553671 306156 0x55a56659d6a0 DEBUG gldebug
>> gstgldebug.c:319:_gst_gl_debug_callback:<glcontextegl0> medium:
>> GL other from API id:1, FBO incomplete: color attachment
>> incomplete [0]
>>
>> (testegl1:306156): GStreamer-CRITICAL **: 15:39:40.709:
>> gst_debug_log_valist: assertion 'category != NULL' failed
>> 0:00:14.023741217 306156 0x55a56659d6a0 WARN glbasetexture
>> gstglmemory.c:404:gst_gl_memory_read_pixels: Could not create
>> framebuffer to read pixels for memory 0x7f10d08992b0
>> 0:00:14.023776145 306156 0x55a56659d6a0 LOG glbasememory
>> gstglbasememory.c:348:_unmap_data_gl: unmapping mem
>> 0x7f105c227d00 flags 20002
>> 0:00:14.023800095 306156 0x55a56659d6a0 LOG glbuffer
>> gstglbuffer.c:305:_gl_buffer_unmap: unmapping 0x7f105c227d00 id 1
>> size 360000
>> 0:00:14.024028993 306156 0x55a56659d6a0 LOG glbasememory
>> gstglbasememory.c:269:_map_data_gl: mapping mem 0x7f10d08992b0
>> flags 10001
>> 0:00:14.024074629 306156 0x55a56659d6a0 LOG glbasememory
>> gstglbasememory.c:227:gst_gl_base_memory_alloc_data:
>> 0x7f10d08992b0 attempting allocation of data pointer of size 360007
>> 0:00:14.024139538 306156 0x55a56659d6a0 DEBUG glbasememory
>> gstglbasememory.c:236:gst_gl_base_memory_alloc_data:
>> 0x7f10d08992b0 allocated data pointer alloc 0x7f10c05fa010, data
>> 0x7f10c05fa010
>> 0:00:14.024169028 306156 0x55a56659d6a0 LOG glbasememory
>> gstglbasememory.c:269:_map_data_gl: mapping mem 0x7f105c227d00
>> flags 20002
>> 0:00:14.024195194 306156 0x55a56659d6a0 LOG glbasememory
>> gstglbasememory.c:269:_map_data_gl: mapping mem 0x7f105c227d00
>> flags 20002
>> 0:00:14.024217673 306156 0x55a56659d6a0 LOG glbasememory
>> gstglbasememory.c:278:_map_data_gl: multiple map no 2 flags 20002
>> all flags 20002
>> 0:00:14.024263516 306156 0x55a56659d6a0 DEBUG gldebug
>> gstgldebug.c:319:_gst_gl_debug_callback:<glcontextegl0> medium:
>> GL other from API id:1, FBO incomplete: color attachment
>> incomplete [0]
>>
>> On Mon, Aug 16, 2021 at 6:55 PM Matthew Waters
>> <ystreet00 at gmail.com <mailto:ystreet00 at gmail.com>> wrote:
>>
>> If you're running on linux, there is a native display handle
>> behind the EGLDIsplay. That native display handle would be
>> an X11 Display, or a wayland wl_display (or maybe a mir
>> display). OpenGL resources cannot be shared unless this
>> 'native' display type is the same between all OpenGL uses.
>> Passing an EGLDisplay is not sufficient and will not work.
>>
>> You need to pass the state->xdisplay using the appropriate
>> platform-specific gst_gl_display_x11_new_with_display(). You
>> cannot use the EGLDisplay version here.
>>
>> Cheers
>> -Matt
>>
>> On 17/8/21 12:40 am, Lusine Hayrapetyan wrote:
>>> Hi Mat,
>>>
>>> My display is *EGLDisplay *type:
>>> /* get an EGL display connection */
>>> EGLNativeDisplayType nativeDisplay = state->xdisplay;
>>> *state->display* = eglGetDisplay (nativeDisplay);
>>> assert (state->display != EGL_NO_DISPLAY);
>>>
>>> /* The I create an EGL rendering context with EGLDisplay */
>>> state->context =
>>> eglCreateContext (state->display, config,
>>> EGL_NO_CONTEXT, context_attributes);
>>>
>>> /* Then I create GstGLDisplayEGL from EGLDisplay */
>>> state->gst_display = gst_gl_display_egl_new_with_egl_display
>>> (state->display);
>>>
>>> /* And create my wrapped context with GstGLDisplayEGL */
>>> state->gl_context =
>>> gst_gl_context_new_wrapped (GST_GL_DISPLAY
>>> (state->gst_display),
>>> (guintptr) state->context, GST_GL_PLATFORM_EGL,
>>> GST_GL_API_GLES2);
>>>
>>> /* Now I want to create a shared context from my wrapped
>>> context and again I pass GstGLDisplayEGL to it */
>>> gst_gl_display_create_context(GST_GL_DISPLAY(state->gst_display),
>>> state->gl_context, &state->newContext, &error) )
>>>
>>> And this function complains that it expects an x11 display
>>> instead of egl display. But I don't understand how I can
>>> pass x11 display because my context is created with egl display.
>>> I'm not experienced in opengl es & egl, do I miss something?
>>>
>>> Regards,
>>> Lusine
>>>
>>> On Fri, Aug 13, 2021 at 1:48 PM Matthew Waters
>>> <ystreet00 at gmail.com <mailto:ystreet00 at gmail.com>> wrote:
>>>
>>>
>>>
>>> On 13/8/21 5:59 pm, Lusine Hayrapetyan wrote:
>>>> In gstglconfig.h I see the following:
>>>> /#define GST_GL_HAVE_WINDOW_X11 1
>>>> /
>>>>
>>>> I've built the latest master on Ubuntu 20 and still see
>>>> the same message about x11 window and there is a new
>>>> FIXME message:
>>>> 0:00:03.382786440 252231 0x55e6075596a0 INFO glwindow
>>>> gstglwindow.c:295:gst_gl_window_new: creating a window,
>>>> user choice:x11
>>>> 0:00:03.382847275 252231 0x55e6075596a0 INFO glwindow
>>>> gstglwindow_x11.c:136:gst_gl_window_x11_new: Wrong
>>>> display type 32 for this window type 1
>>>
>>> This is the problematic line. The display type (EGL) as
>>> returned by gst_gl_display_get_type() does not match the
>>> window impl's expectations (X11). If you are running
>>> under X11 and your application is using X11 to connect
>>> to the display server, you need to pass an X11
>>> GstGLDisplay. Passing an EGLDisplay is not sufficient
>>> and will not work. An appropriate EGLDIsplay will be
>>> created internally.
>>>
>>> Cheers
>>> -Matt
>>>
>>>> 0:00:03.382877455 252231 0x55e6075596a0 WARN glwindow
>>>> gstglwindow.c:340:gst_gl_window_new: Could not create
>>>> window. user specified x11, creating dummy window
>>>> 0:00:03.382998854 252231 0x55e6075596a0 DEBUG
>>>> gldisplay
>>>> gstgldisplay.c:694:gst_gl_display_create_window:<gldisplayegl0>
>>>> Adding window <gldummywindow0> (0x55e607797c50) to
>>>> internal list
>>>> 0:00:03.383024753 252231 0x55e6075596a0 DEBUG
>>>> glcontext
>>>> gstglcontext.c:963:gst_gl_context_set_window:<glcontextegl0>
>>>> window:<gldummywindow0>
>>>> 0:00:03.383269595 252231 0x55e6077b5640 DEBUG
>>>> glcontext
>>>> gstglcontext.c:1223:gst_gl_context_create_thread:<glcontextegl0>
>>>> Creating thread
>>>> 0:00:03.383377039 252231 0x55e6077b5640 FIXME
>>>> glcontext
>>>> gstglcontext.c:2039:gst_gl_wrapped_context_get_config:<glwrappedcontext0>
>>>> wrapped context could not retrieve config. The
>>>> application may be missing a call to
>>>> gst_gl_context_fill_info() or the specific platform
>>>> implemention is not implemented for retrieving the
>>>> config from a wrapped OpenGL context.
>>>> 0:00:03.383437907 252231 0x55e6077b5640 INFO glcontext
>>>> gstglcontext.c:1315:gst_gl_context_create_thread:<glcontextegl0>
>>>> Attempting to create opengl context. user chosen api(s)
>>>> (gles2), compiled api support (opengl opengl3 gles2)
>>>> display api (gles2)
>>>> 0:00:03.383463315 252231 0x55e6077b5640 DEBUG
>>>> glcontext
>>>> gstglcontext_egl.c:863:gst_gl_context_egl_create_context:<glcontextegl0>
>>>> Creating EGL context
>>>>
>>>> Regards,
>>>> Lusine
>>>>
>>>> On Thu, Aug 12, 2021 at 11:18 AM Matthew Waters
>>>> <ystreet00 at gmail.com <mailto:ystreet00 at gmail.com>> wrote:
>>>>
>>>> Yes, that is a critical error for X11. X11 would
>>>> require access to an X11 Display which must be
>>>> provided by your application in order share OpenGL
>>>> resources. You may also not have X11 support built
>>>> and you would have to check your build
>>>> configuration for that. gstglconfig.h would contain
>>>> information on what libgstgl-1.0 was built with.
>>>>
>>>> It looks like it's picking a raw EGLDisplay
>>>> instead. This was potentially an issue in previous
>>>> versions of GStreamer that has been fixed in master
>>>> by
>>>> https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/merge_requests/1154
>>>> <https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/merge_requests/1154>
>>>> and
>>>> https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/merge_requests/1169
>>>> <https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/merge_requests/1169>.
>>>>
>>>> Cheers
>>>> -Matt
>>>>
>>>> On 12/8/21 1:50 am, Lusine Hayrapetyan wrote:
>>>>> For limiting to a particular OpenGL variant I'm
>>>>> using the following environment variables, is it
>>>>> the same?
>>>>> / export GST_GL_WINDOW=x11
>>>>> export GST_GL_PLATFORM=egl
>>>>> export GST_GL_API=gles2/
>>>>> /
>>>>> /
>>>>> I've attached GST_DEBUG=gl*:7 output (gl_output.txt).
>>>>> I see the following messages in the output:
>>>>> /0:00:03.857084617 24438 0x55d59121d400 INFO
>>>>> glwindow gstglwindow.c:278:gst_gl_window_new:
>>>>> creating a window, user choice:x11
>>>>> 0:00:03.857117499 24438 0x55d59121d400 INFO
>>>>> glwindow
>>>>> gstglwindow_x11.c:137:gst_gl_window_x11_new: Wrong
>>>>> display type 32 for this window type 1
>>>>> 0:00:03.857144332 24438 0x55d59121d400 WARN
>>>>> glwindow gstglwindow.c:324:gst_gl_window_new:
>>>>> Could not create window. user specified x11,
>>>>> creating dummy window
>>>>> 0:00:03.857277761 24438 0x55d59121d400 DEBUG
>>>>> gldisplay
>>>>> gstgldisplay.c:579:gst_gl_display_create_window:<gldisplayegl0>
>>>>> Adding window <gldummywindow0> (0x55d5913809d0) to
>>>>> internal list
>>>>> 0:00:03.857311159 24438 0x55d59121d400 DEBUG
>>>>> glcontext
>>>>> gstglcontext.c:948:gst_gl_context_set_window:<glcontextegl0>
>>>>> window:<gldummywindow0>/
>>>>> /
>>>>> /
>>>>> Is this a critical issue? Can this cause a texture
>>>>> reading issue?
>>>>>
>>>>> On Tue, Aug 10, 2021 at 10:21 AM Matthew Waters
>>>>> <ystreet00 at gmail.com <mailto:ystreet00 at gmail.com>>
>>>>> wrote:
>>>>>
>>>>> Ah, if you want to limit to a particular
>>>>> OpenGL variant, you need to also call
>>>>> gst_gl_display_filter_gl_api() with that variant.
>>>>>
>>>>> GST_DEBUG=gl*:7 will output a whole bunch of
>>>>> information about what is generated and tried
>>>>> in all of this respect.
>>>>>
>>>>> Cheers
>>>>> -Matt
>>>>>
>>>>> On 9/8/21 12:58 am, Lusine Hayrapetyan wrote:
>>>>>> Hi Matthew,
>>>>>> Seems the second approach fits to my use
>>>>>> case- I need to push textures which are
>>>>>> created in opengles & egl( it means not in
>>>>>> gstreamer context) to gstreamer.
>>>>>> I've changed my code to use
>>>>>> |gst_gl_display_create_context & |
>>>>>> |gst_gl_display_add_context but still can't
>>>>>> read textures in gstreamer, gstreamer
>>>>>> prudeces the following errors after I push
>>>>>> buffer to appsrc:|
>>>>>> |(testegl1:24602): GStreamer-CRITICAL **:
>>>>>> 13:02:55.568: gst_debug_log_valist: assertion
>>>>>> 'category != NULL' failed
>>>>>> 0:00:14.039444188 24602 0x5611bc356d90 WARN
>>>>>> glbasetexture
>>>>>> gstglmemory.c:401:gst_gl_memory_read_pixels:
>>>>>> Could not create framebuffer to read pixels
>>>>>> for memory 0x7f8cf0017ac0
>>>>>> 0:00:14.039480909 24602 0x5611bc356d90 WARN
>>>>>> glbasememory
>>>>>> gstglbasememory.c:585:gst_gl_base_memory_memcpy:
>>>>>> could not read map source memory 0x7f8cf0017ac0
>>>>>> 0:00:14.039505741 24602 0x5611bc356d90 WARN
>>>>>> glmemory gstglmemorypbo.c:592:_gl_mem_copy:
>>>>>> Could not copy GL Memory
>>>>>> 0:00:14.039779268 24602 0x5611bc4f14f0 ERROR
>>>>>> videometa gstvideometa.c:247:default_map:
>>>>>> cannot map memory range 0-1
>>>>>> 0:00:14.039846056 24602 0x5611bc4f14f0 ERROR
>>>>>> default
>>>>>> video-frame.c:168:gst_video_frame_map_id:
>>>>>> failed to map video frame plane 0
>>>>>> 0:00:14.039891314 24602 0x5611bc4f14f0 WARN
>>>>>> videofilter
>>>>>> gstvideofilter.c:297:gst_video_filter_transform:<videoconvert0>
>>>>>> warning: invalid video buffer received
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> |
>>>>>> |This is how I implemented it, did I
>>>>>> misunderstand something? |
>>>>>> |
>>>>>> |
>>>>>> I have rendering thread where I initialize
>>>>>> opengles context and create wrapped and the
>>>>>> new contexts.
>>>>>> //
>>>>>> // Description: Sets the display, OpenGL|ES
>>>>>> context and screen stuff
>>>>>> // Created GstGLContext s - wrapped
>>>>>> (gst_gl_context_new_wrapped) and new
>>>>>> context(gst_gl_display_create_context)
>>>>>> //
>>>>>> static void
>>>>>> init_ogl (APP_STATE_T * state)
>>>>>> {
>>>>>> ...
>>>>>> /* get an EGL display connection */
>>>>>> state->display = eglGetDisplay
>>>>>> (EGL_DEFAULT_DISPLAY);
>>>>>> assert (state->display != EGL_NO_DISPLAY);
>>>>>> /* initialize the EGL display connection */
>>>>>> result = eglInitialize (state->display,
>>>>>> NULL, NULL);
>>>>>> assert (EGL_FALSE != result);
>>>>>>
>>>>>> /* create an EGL rendering context */
>>>>>> state->context =
>>>>>> eglCreateContext (state->display, config,
>>>>>> EGL_NO_CONTEXT, context_attributes);
>>>>>> assert (state->context != EGL_NO_CONTEXT);
>>>>>> //
>>>>>> // Initialize GStreamer related resources.
>>>>>> //
>>>>>> state->gst_display =
>>>>>> gst_gl_display_egl_new_with_egl_display
>>>>>> (state->display);
>>>>>> state->gl_context =
>>>>>> gst_gl_context_new_wrapped (GST_GL_DISPLAY
>>>>>> (state->gst_display),
>>>>>> (guintptr) state->context,
>>>>>> GST_GL_PLATFORM_EGL, GST_GL_API_GLES2);
>>>>>>
>>>>>> GError *error = NULL;
>>>>>> if (
>>>>>> !gst_gl_display_create_context(GST_GL_DISPLAY(state->gst_display),
>>>>>> state->gl_context, &state->newContext, &error) )
>>>>>> g_print("Failed to create new context\n");
>>>>>>
>>>>>> if (
>>>>>> !gst_gl_display_add_context(GST_GL_DISPLAY(state->gst_display),
>>>>>> state->newContext))
>>>>>> g_print("Failed to add new context to
>>>>>> display\n");
>>>>>>
>>>>>> } // init_ogl end.
>>>>>>
>>>>>>
>>>>>> static void
>>>>>> sync_bus_call (GstBus * bus, GstMessage *
>>>>>> msg, gpointer * data)
>>>>>> {
>>>>>>
>>>>>> APP_STATE_T *state = (APP_STATE_T *)data;
>>>>>> switch (GST_MESSAGE_TYPE (msg))
>>>>>> {
>>>>>> case GST_MESSAGE_NEED_CONTEXT:
>>>>>> {
>>>>>> const gchar *context_type;
>>>>>> gst_message_parse_context_type (msg,
>>>>>> &context_type);
>>>>>>
>>>>>> GstContext *context = NULL;
>>>>>> if (g_strcmp0 (context_type,
>>>>>> GST_GL_DISPLAY_CONTEXT_TYPE) == 0)
>>>>>> {
>>>>>> GstGLDisplay * gl_display =
>>>>>> GST_GL_DISPLAY(state->gst_display);
>>>>>> context = gst_context_new
>>>>>> (GST_GL_DISPLAY_CONTEXT_TYPE, TRUE);
>>>>>> gst_context_set_gl_display(context, gl_display);
>>>>>> gst_element_set_context
>>>>>> (GST_ELEMENT(msg->src), context);
>>>>>> }
>>>>>> else if (g_strcmp0 (context_type,
>>>>>> "gst.gl.app_context") == 0)
>>>>>> {
>>>>>> GstContext *context =
>>>>>> gst_context_new("gst.gl.app_context", TRUE);
>>>>>> GstStructure *s =
>>>>>> gst_context_writable_structure (context);
>>>>>> gst_structure_set (s, "context",
>>>>>> GST_TYPE_GL_CONTEXT, state->gl_context, NULL);
>>>>>> gst_element_set_context(GST_ELEMENT(msg->src),
>>>>>> context);
>>>>>> }
>>>>>> break;
>>>>>> }
>>>>>> default:
>>>>>> break;
>>>>>> }
>>>>>> } // sync_bus_call end
>>>>>>
>>>>>>
>>>>>> I use need-data callback to create a buffer
>>>>>> from texture_id and and push it in the appsrc:
>>>>>> /g_signal_connect (state->appsrc,
>>>>>> "need-data", G_CALLBACK (/ pushFrame /), state);/
>>>>>> /
>>>>>> /
>>>>>> static bool pushFrame(..., APP_STATE_T * state)
>>>>>> {
>>>>>> // Wrap the texture into GstGLMemory
>>>>>> GstVideoInfo vinfo;
>>>>>> gst_video_info_set_format(&vinfo,
>>>>>> GST_VIDEO_FORMAT_RGBA, state->screen_width,
>>>>>> state->screen_height);
>>>>>> // Use state->newContext for allocator.
>>>>>> GstAllocator* allocator =
>>>>>> GST_ALLOCATOR(gst_gl_memory_allocator_get_default(state->newContext));
>>>>>> GstGLVideoAllocationParams* params =
>>>>>> gst_gl_video_allocation_params_new_wrapped_texture(
>>>>>> state->newContext, NULL, &vinfo, 0, NULL,
>>>>>> GST_GL_TEXTURE_TARGET_2D, GST_GL_RGBA,
>>>>>> state->tex,
>>>>>> NULL, 0);
>>>>>>
>>>>>> GstGLMemory* glMemory =
>>>>>> GST_GL_MEMORY_CAST(gst_gl_base_memory_alloc(
>>>>>> GST_GL_BASE_MEMORY_ALLOCATOR_CAST(allocator),
>>>>>> (GstGLAllocationParams*) params));
>>>>>>
>>>>>> gst_gl_allocation_params_free((GstGLAllocationParams
>>>>>> *)params);
>>>>>> gst_object_unref(allocator);
>>>>>>
>>>>>> // Attach GstGLMemory object into buffer,
>>>>>> timestamp the buffer and push it downstream
>>>>>> GstBuffer* buffer = gst_buffer_new();
>>>>>> gst_buffer_append_memory(buffer,
>>>>>> GST_MEMORY_CAST(glMemory));
>>>>>>
>>>>>> GstVideoMeta * vmeta =
>>>>>> gst_buffer_add_video_meta(buffer,
>>>>>> GST_VIDEO_FRAME_FLAG_NONE,
>>>>>> GST_VIDEO_FORMAT_RGBA, state->screen_width,
>>>>>> state->screen_height);
>>>>>>
>>>>>> // Put timestamps into buffer
>>>>>> GST_BUFFER_PTS (buffer) = timestamp;
>>>>>> GST_BUFFER_DURATION (buffer) =
>>>>>> gst_util_uint64_scale_int (1, GST_SECOND, 2);
>>>>>> timestamp += GST_BUFFER_DURATION (buffer);
>>>>>>
>>>>>> GstFlowReturn ret;
>>>>>> g_signal_emit_by_name(state->appsrc,
>>>>>> "push-buffer", buffer, &ret);
>>>>>>
>>>>>> if (ret != GST_FLOW_OK)
>>>>>> {
>>>>>> // Something wrong, stop pushing.
>>>>>> g_printerr("Something went wrong: Pushing
>>>>>> buffer into appsrc is stopped.\n");
>>>>>> return false;
>>>>>> }
>>>>>>
>>>>>> return true;
>>>>>> } // pushFrame end
>>>>>>
>>>>>> Regards,
>>>>>> Lusine
>>>>>>
>>>>>> On Fri, Aug 6, 2021 at 8:59 AM Matthew Waters
>>>>>> <ystreet00 at gmail.com
>>>>>> <mailto:ystreet00 at gmail.com>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On 6/8/21 1:12 am, Lusine Hayrapetyan wrote:
>>>>>>> Dear Matt,
>>>>>>> Thank you very much for your response.
>>>>>>> It helped me to understand that using
>>>>>>> 'wrapped' OpenGL context is a wrong
>>>>>>> direction to go :)
>>>>>>>
>>>>>>> /WRT Suggestion 1:/
>>>>>>> Do I understand correctly that I need to
>>>>>>> get local context from gstreamer and
>>>>>>> pass it to opengl rendering thread? (I
>>>>>>> have a rendering thread where I set
>>>>>>> OpenGL|ES context and screen stuff).
>>>>>>> GstGLContext* mContext = nullptr;
>>>>>>> g_object_get(state->gldownload,
>>>>>>> "context", &mContext, NULL);
>>>>>>> guintptr handle;
>>>>>>> handle =
>>>>>>> gst_gl_context_get_gl_context(mContext);
>>>>>>> // is this correct?
>>>>>>> state->context = (EGLContext)handle; //
>>>>>>> state->context is EGLContext type;
>>>>>>> And then use state->context in OpenGL|ES?
>>>>>>> Do I need to get and pass window and
>>>>>>> display from gstreamer to my rendering
>>>>>>> thread as well?
>>>>>>> Although my use scenario is different
>>>>>>> from this one - I need to pass context
>>>>>>> from OpenGL to Gstreamer.
>>>>>>
>>>>>> You only need to retrieve or create a
>>>>>> non-wrapped GstGLContext and use that for
>>>>>> creating your textures that you are
>>>>>> pushing into GStreamer. You don't need
>>>>>> to use GStreamer's provided OpenGL
>>>>>> context for anything else. Everything
>>>>>> else in your sample remains the same. You
>>>>>> may need to add a GstGLSyncMeta on your
>>>>>> buffers you are pushing into GStreamer to
>>>>>> provide the necessary synchronisation
>>>>>> guarantees between the shared OpenGL
>>>>>> contexts (application and GStreamer). On
>>>>>> some platforms the window handle type and
>>>>>> format may be important however in
>>>>>> general on linux (X11/wayland) it doesn't
>>>>>> really matter.
>>>>>>
>>>>>> You must not attempt to use GStreamer's
>>>>>> OpenGL context as-is (using e.g.
>>>>>> eglMakeCurrent() or anything of the like)
>>>>>> from outside the GStreamer OpenGL context
>>>>>> thread (as provided by the
>>>>>> gst_gl_context_thread_add() API).
>>>>>>
>>>>>>> /WRT Suggestion 2:/
>>>>>>> gst_gl_display_create_context accepts
>>>>>>> /other_context /argument, should the
>>>>>>> /other_context /be the 'wrapped' context?
>>>>>>
>>>>>> Yes. other_context is the GstGLContext
>>>>>> that will be shared with the newly
>>>>>> created GstGLContext.
>>>>>>
>>>>>> Side note, GStreamer cannot use any
>>>>>> application-provided OpenGL context as-is
>>>>>> due to the overhead of dealing with all
>>>>>> the OpenGL state that may be changed
>>>>>> behind GStreamer's back. This is why the
>>>>>> OpenGL context sharing dance is required.
>>>>>>
>>>>>> Cheers
>>>>>> -Matt
>>>>>>
>>>>>>> Best Regards,
>>>>>>> Lusine
>>>>>>>
>>>>>>> On Thu, Aug 5, 2021 at 12:39 PM Matthew
>>>>>>> Waters <ystreet00 at gmail.com
>>>>>>> <mailto:ystreet00 at gmail.com>> wrote:
>>>>>>>
>>>>>>> So, I think you almost have the
>>>>>>> correct sequence.
>>>>>>>
>>>>>>> Response inline.
>>>>>>>
>>>>>>> On 5/8/21 1:04 am, Lusine
>>>>>>> Hayrapetyan via gstreamer-devel wrote:
>>>>>>>> Hi Folks,
>>>>>>>> I'm struggling with the following
>>>>>>>> issue and can't understand what I'm
>>>>>>>> doing wrong.
>>>>>>>> I need to pass opengl texture to
>>>>>>>> the gstreamer pipeline.
>>>>>>>> I have a rendering thread where I
>>>>>>>> create opengl texture, the
>>>>>>>> following objects created in this
>>>>>>>> thread:
>>>>>>>> /EGLDisplay display;/
>>>>>>>> /EGLContext context;/
>>>>>>>> /
>>>>>>>> /
>>>>>>>> I create gstreamer pipeline in the
>>>>>>>> main thread and as described in the
>>>>>>>> following article sharing an X11
>>>>>>>> display and GstGLContext with the
>>>>>>>> bus callback.
>>>>>>>> http://ystreet00.blogspot.com/2015/09/gstreamer-16-and-opengl-contexts.html
>>>>>>>> <http://ystreet00.blogspot.com/2015/09/gstreamer-16-and-opengl-contexts.html>
>>>>>>>>
>>>>>>>> GstGLDisplayEGL and GstGLContext
>>>>>>>> are created in this way:
>>>>>>>> /GstGLDisplayEGL* gst_display =
>>>>>>>> gst_gl_display_egl_new_with_egl_display
>>>>>>>> (display);
>>>>>>>> /
>>>>>>>> /GstGLContext *gl_context =
>>>>>>>> gst_gl_context_new_wrapped
>>>>>>>> (GST_GL_DISPLAY (gst_display),
>>>>>>>> (guintptr) context,
>>>>>>>> GST_GL_PLATFORM_EGL,
>>>>>>>> GST_GL_API_GLES2);/
>>>>>>>>
>>>>>>>> The first element of my pipeline is
>>>>>>>> appsrc:
>>>>>>>> /appsrc stream-type=0
>>>>>>>> emit-signals=1 format=3
>>>>>>>> caps=video/x-raw(memory:GLMemory), width=300,
>>>>>>>> height=300,
>>>>>>>> framerate=(fraction)20/1,
>>>>>>>> format=(string)RGBA ! gldownload ! .../
>>>>>>>>
>>>>>>>> I use need-data callback to create
>>>>>>>> a buffer from texture_id and and
>>>>>>>> push it in the appsrc:
>>>>>>>> /g_signal_connect (state->appsrc,
>>>>>>>> "need-data", G_CALLBACK (/
>>>>>>>> pushFrame /), state);/
>>>>>>>> /
>>>>>>>> /
>>>>>>>> /bool pushFrame()/
>>>>>>>> /{
>>>>>>>> // Wrap the texture into
>>>>>>>> GstGLMemory
>>>>>>>> GstVideoInfo vinfo;
>>>>>>>> gst_video_info_set_format(&vinfo,
>>>>>>>> GST_VIDEO_FORMAT_RGBA, 300, 300);
>>>>>>>>
>>>>>>>> GstAllocator* allocator =
>>>>>>>> GST_ALLOCATOR(gst_gl_memory_allocator_get_default(gl_context));
>>>>>>>>
>>>>>>>> GstGLVideoAllocationParams* params
>>>>>>>> =
>>>>>>>> gst_gl_video_allocation_params_new_wrapped_texture(
>>>>>>>> state->gl_context, NULL, &vinfo,
>>>>>>>> 0, NULL, GST_GL_TEXTURE_TARGET_2D,
>>>>>>>> GST_GL_RGBA, / texture_id /,
>>>>>>>> NULL, 0);
>>>>>>>> /
>>>>>>>
>>>>>>> The use of state->gl_context is
>>>>>>> probably your OpenGL context that
>>>>>>> has been wrapped from the
>>>>>>> application. This 'wrapped' OpenGL
>>>>>>> context has some limitations, one
>>>>>>> being that GStreamer cannot actually
>>>>>>> do a complete
>>>>>>> gst_gl_context_thread_add where the
>>>>>>> request is marshalled to an
>>>>>>> OpenGL-specific thread. This is
>>>>>>> what that critical is complaining
>>>>>>> about effectively.
>>>>>>>
>>>>>>> To do this properly, you would need
>>>>>>> to do one of two things:
>>>>>>> 1. Retrieve the OpenGL context from
>>>>>>> the downstream gldownload element
>>>>>>> using either the 'context' property
>>>>>>> or using an appropriate GST_CONTEXT
>>>>>>> QUERY or the helper
>>>>>>> gst_gl_query_local_gl_context().
>>>>>>> 2. Create your own GStreamer OpenGL
>>>>>>> context and add it to the
>>>>>>> GstGLDisplay using something like:
>>>>>>> https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/blob/master/gst-libs/gst/gl/gstglbasefilter.c#L550-565
>>>>>>> <https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/blob/master/gst-libs/gst/gl/gstglbasefilter.c#L550-565>.
>>>>>>>
>>>>>>> Cheers
>>>>>>> -Matt
>>>>>>>
>>>>>>>> /
>>>>>>>> // The following line produces
>>>>>>>> an error!!!
>>>>>>>> GstGLMemory* glMemory =
>>>>>>>> GST_GL_MEMORY_CAST(gst_gl_base_memory_alloc(
>>>>>>>> GST_GL_BASE_MEMORY_ALLOCATOR_CAST(allocator),
>>>>>>>> (GstGLAllocationParams*) params));
>>>>>>>>
>>>>>>>> gst_gl_allocation_params_free((GstGLAllocationParams
>>>>>>>> *)params);
>>>>>>>> gst_object_unref(allocator);
>>>>>>>>
>>>>>>>> // Attach GstGLMemory object
>>>>>>>> into buffer, timestamp the buffer
>>>>>>>> and push it downstream
>>>>>>>> GstBuffer* buffer =
>>>>>>>> gst_buffer_new();
>>>>>>>> gst_buffer_append_memory(buffer,
>>>>>>>> GST_MEMORY_CAST(glMemory));
>>>>>>>>
>>>>>>>> // Put timestamps into buffer
>>>>>>>> GST_BUFFER_PTS (buffer) = timestamp;
>>>>>>>> GST_BUFFER_DURATION (buffer) =
>>>>>>>> gst_util_uint64_scale_int (1,
>>>>>>>> GST_SECOND, 2);
>>>>>>>>
>>>>>>>> timestamp +=
>>>>>>>> GST_BUFFER_DURATION (buffer);
>>>>>>>> GstFlowReturn ret;
>>>>>>>> g_signal_emit_by_name(state->appsrc,
>>>>>>>> "push-buffer", buffer, &ret);
>>>>>>>>
>>>>>>>> if (ret != GST_FLOW_OK)
>>>>>>>> {
>>>>>>>> // Something wrong, stop
>>>>>>>> pushing.
>>>>>>>> g_printerr("Something went wrong:
>>>>>>>> Pushing buffer into appsrc is
>>>>>>>> stopped.\n");
>>>>>>>> return false;
>>>>>>>> }
>>>>>>>>
>>>>>>>> return true;
>>>>>>>> }
>>>>>>>> /
>>>>>>>>
>>>>>>>> pushFrame produces the following error:
>>>>>>>> gst_gl_context_thread_add:
>>>>>>>> assertion
>>>>>>>> 'context->priv->active_thread ==
>>>>>>>> g_thread_self ()' failIed
>>>>>>>>
>>>>>>>> What am I doing wrong or how can
>>>>>>>> push gpu texture to gstreamer?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Lusine
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210818/dc773768/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210818/dc773768/attachment-0001.sig>
More information about the gstreamer-devel
mailing list