push opengl texture to gstreamer pipeline

[Matthew Waters]

Hi,

On 6/8/21 1:12 am, Lusine Hayrapetyan wrote:
> Dear Matt,
> Thank you very much for your response. It helped me to understand that 
> using  'wrapped' OpenGL context is a wrong direction to go :)
>
> /WRT Suggestion 1:/
> Do I understand correctly that I need to get local context from 
> gstreamer and pass it to opengl rendering thread? (I have a rendering 
> thread where I set OpenGL|ES context and screen stuff).
> GstGLContext* mContext = nullptr;
> g_object_get(state->gldownload, "context", &mContext, NULL);
> guintptr handle;
> handle = gst_gl_context_get_gl_context(mContext); // is this correct?
> state->context = (EGLContext)handle; // state->context is EGLContext type;
> And then use state->context in OpenGL|ES?
> Do I need to get and pass window and display from gstreamer to my 
> rendering thread as well?
> Although my use scenario is different from this one - I need to pass 
> context from OpenGL to Gstreamer.

You only need to retrieve or create a non-wrapped GstGLContext and use 
that for creating your textures that you are pushing into GStreamer.  
You don't need to use GStreamer's provided OpenGL context for anything 
else.  Everything else in your sample remains the same.  You may need to 
add a GstGLSyncMeta on your buffers you are pushing into GStreamer to 
provide the necessary synchronisation guarantees between the shared 
OpenGL contexts (application and GStreamer).  On some platforms the 
window handle type and format may be important however in general on 
linux (X11/wayland) it doesn't really matter.

You must not attempt to use GStreamer's OpenGL context as-is (using e.g. 
eglMakeCurrent() or anything of the like) from outside the GStreamer 
OpenGL context thread (as provided by the gst_gl_context_thread_add() API).

> /WRT Suggestion 2:/
> gst_gl_display_create_context accepts /other_context /argument, should 
> the /other_context /be the 'wrapped' context?

Yes.  other_context is the GstGLContext that will be shared with the 
newly created GstGLContext.

Side note, GStreamer cannot use any application-provided OpenGL context 
as-is due to the overhead of dealing with all the OpenGL state that may 
be changed behind GStreamer's back.  This is why the OpenGL context 
sharing dance is required.

Cheers
-Matt

> Best Regards,
> Lusine
>
> On Thu, Aug 5, 2021 at 12:39 PM Matthew Waters <ystreet00 at gmail.com 
> <mailto:ystreet00 at gmail.com>> wrote:
>
>     So, I think you almost have the correct sequence.
>
>     Response inline.
>
>     On 5/8/21 1:04 am, Lusine Hayrapetyan via gstreamer-devel wrote:
>>     Hi Folks,
>>     I'm struggling with the following issue and can't understand what
>>     I'm doing wrong.
>>     I need to pass opengl texture to the gstreamer pipeline.
>>     I have a rendering thread where I create opengl texture, the
>>     following objects created in this thread:
>>     /EGLDisplay display;/
>>     /EGLContext context;/
>>     /
>>     /
>>     I create gstreamer pipeline in the main thread and as described
>>     in the following article sharing an X11 display and GstGLContext
>>     with the bus callback.
>>     http://ystreet00.blogspot.com/2015/09/gstreamer-16-and-opengl-contexts.html
>>     <http://ystreet00.blogspot.com/2015/09/gstreamer-16-and-opengl-contexts.html>
>>
>>     GstGLDisplayEGL and GstGLContext are created in this way:
>>     /GstGLDisplayEGL* gst_display =
>>     gst_gl_display_egl_new_with_egl_display (display);
>>     /
>>     /GstGLContext *gl_context =
>>           gst_gl_context_new_wrapped (GST_GL_DISPLAY (gst_display),
>>           (guintptr) context, GST_GL_PLATFORM_EGL, GST_GL_API_GLES2);/
>>
>>     The first element of my pipeline is appsrc:
>>     /appsrc stream-type=0 emit-signals=1 format=3
>>     caps=video/x-raw(memory:GLMemory),  width=300, height=300,
>>     framerate=(fraction)20/1, format=(string)RGBA ! gldownload ! .../
>>
>>     I use need-data callback to create a buffer from texture_id and
>>     and push it in the appsrc:
>>     /g_signal_connect (state->appsrc, "need-data", G_CALLBACK (/
>>     pushFrame /), state);/
>>     /
>>     /
>>     /bool pushFrame()/
>>     /{
>>          // Wrap the texture into GstGLMemory
>>         GstVideoInfo vinfo;
>>         gst_video_info_set_format(&vinfo, GST_VIDEO_FORMAT_RGBA, 300,
>>     300);
>>
>>         GstAllocator* allocator =
>>     GST_ALLOCATOR(gst_gl_memory_allocator_get_default(gl_context));
>>
>>         GstGLVideoAllocationParams* params =
>>     gst_gl_video_allocation_params_new_wrapped_texture(
>>            state->gl_context, NULL, &vinfo, 0, NULL,
>>     GST_GL_TEXTURE_TARGET_2D, GST_GL_RGBA, / texture_id /,
>>           NULL, 0);
>>     /
>
>     The use of state->gl_context is probably your OpenGL context that
>     has been wrapped from the application.  This 'wrapped' OpenGL
>     context has some limitations, one being that GStreamer cannot
>     actually do a complete gst_gl_context_thread_add where the request
>     is marshalled to an OpenGL-specific thread.  This is what that
>     critical is complaining about effectively.
>
>     To do this properly, you would need to do one of two things:
>     1. Retrieve the OpenGL context from the downstream gldownload
>     element using either the 'context' property or using an
>     appropriate GST_CONTEXT QUERY or the helper
>     gst_gl_query_local_gl_context().
>     2. Create your own GStreamer OpenGL context and add it to the
>     GstGLDisplay using something like:
>     https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/blob/master/gst-libs/gst/gl/gstglbasefilter.c#L550-565
>     <https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/blob/master/gst-libs/gst/gl/gstglbasefilter.c#L550-565>.
>
>     Cheers
>     -Matt
>
>>     /
>>         // The following line produces an error!!!
>>         GstGLMemory* glMemory =
>>     GST_GL_MEMORY_CAST(gst_gl_base_memory_alloc(
>>           GST_GL_BASE_MEMORY_ALLOCATOR_CAST(allocator),
>>     (GstGLAllocationParams*) params));
>>
>>     gst_gl_allocation_params_free((GstGLAllocationParams *)params);
>>         gst_object_unref(allocator);
>>
>>         // Attach GstGLMemory object into buffer, timestamp the
>>     buffer and push it downstream
>>         GstBuffer* buffer = gst_buffer_new();
>>         gst_buffer_append_memory(buffer, GST_MEMORY_CAST(glMemory));
>>
>>         // Put timestamps into buffer
>>         GST_BUFFER_PTS (buffer) = timestamp;
>>         GST_BUFFER_DURATION (buffer) = gst_util_uint64_scale_int (1,
>>     GST_SECOND, 2);
>>
>>         timestamp += GST_BUFFER_DURATION (buffer);
>>         GstFlowReturn ret;
>>         g_signal_emit_by_name(state->appsrc, "push-buffer", buffer,
>>     &ret);
>>
>>         if (ret != GST_FLOW_OK)
>>         {
>>             // Something wrong, stop pushing.
>>             g_printerr("Something went wrong: Pushing buffer into
>>     appsrc is stopped.\n");
>>             return false;
>>         }
>>
>>         return true;
>>     }
>>     /
>>
>>     pushFrame produces the following error:
>>     gst_gl_context_thread_add: assertion
>>     'context->priv->active_thread == g_thread_self ()' failIed
>>
>>     What am I doing wrong or how can push gpu texture to gstreamer?
>>
>>     Thanks,
>>     Lusine
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210806/a7182e48/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210806/a7182e48/attachment.sig>