[gst-devel] video sink to update OpenGL texture?

Tue Mar 24 22:41:41 CET 2009

Hi,

2009/3/24 Andres Colubri <andres.colubri at gmail.com>

> Hi, I have been following this discussion with interest and I just
> wanted to add a couple of comments/questions.
>
> I'm currently combining opengl and gstreamer within my application.
> Right now, the gst-originated video buffers are pushed out of the
> pipeline with a handoff signal, and then uploaded to an opengl texture
> for further gpu-accelerated processing and final display on the
> application window.

ok

>
>
> But it would be quite great if I can just get the gl texture id from the
> gst-plugins, for instance glupload, and then directly use the texture
> inside my application.

Would be cool but it's not possible for the reasons you explain next. (a
texture id can be used only inside the opengl context where it comes from)

But there is maybe an other way:
--on win32:
wglCopyContext(glcontextSrc, glcontextDst, GL_TEXTURE_BIT);
--on linux (assume it's X):
glXCopyContext(disp, glcontextSrc, glcontextDst, GL_TEXTURE_BIT);
--on OpenGL ES 2.0:
http://www.khronos.org/opengles/sdk/1.1/docs/man/eglCreateContext.xml
(Specifies the EGL rendering context with which to share texture objects)

I have not experimented those things a lot. So I cannot tell more about this
possible way. (there is some restrictions but in our case we are ok)
**
So we can try it, for example I enable a function from a glfilter that would
allow to retrieve the glcontext used inside the pipeline.
You could then just call glCopyContext one time, and then use the texture
id(s) just like you want.

>
> Since my app is cross-platform (linux, win, osx), the composite
> redirection approach doesn't seem viable. The XOverlay interface won't
> work either, because I need to do some gpgpu post-processing before the
> actual image rendering.

Yup, the XOverlay interface is only there to specify a window id. There is
not any relation with OpenGL (I mean about the goal).

  From what I read, one alternative (perhaps the only one under this type
> of scenarios) would be to share the opengl context between the app and
> the gst gl-plugin. Basically, the app would create the opengl context
> and then pass it to the plugin when initializing the gst pipeline.

This is a solution (use the same opengl context bettween you app and the
plugin).
I also first added a function but it's currently not implemented because of
time (and I think there is a better/prefered way).
http://cgit.freedesktop.org/gstreamer/gst-plugins-gl/tree/gst-libs/gst/gl/gstglwindow.h
http://cgit.freedesktop.org/gstreamer/gst-plugins-gl/tree/gst-libs/gst/gl

The main bad thing would be to use a lot of glMakeCurrent call (for each
frame),  inside your app thread and inside the gst-gl thread.
For now, there is only one call to glMakeCurrent at gst-gl init.
Switching a gl context bettween different threads it's very expensive. More
over with several opengl context (several branch in the pipeline)
this solution would be like a hell. I have experience on this so I can tell
more and that's a bad way.

Is this possible in the current implementation of gst-plugins? If not,
> wouldn't be an useful (additional) method for setting up gl-accelerated
> pipelines?

There is 2 other solutions.

First, have you take a look at glfilterapp ?
Look at the code of
gst-plugins-gl/tests/examples/generic/recordgraphic/main.cpp
There is a drawcallback, which is invoked each time a new video frame is
upload, and at the point of glfilterapp place in the graph.
This is the easiest way, if your scene is not extremelly complex.
In this way, you have not to create/handle the opengl context, but you can
give a windows ID in which you want see the gl scene.

The last solution is to write a glfilter as glbumper, gleffects,
glfiltercube, glfiltersobel (etc..) are, and use you animation API inside
the implementation of its.
Also you could control your filter through gobject properties as it's done
for gleffects (see property "effect").
And it would be more flexible.
Finally, you could also split your rendering process in several glfilters.
For example, one for the scene (rotation, quad), and one other for the
effect on the texture.
In this way too, you have not to create/handle the opengl context, but you
can give a windows ID in which you want see the gl scene.
I think it's the best way, the more viable, useable, flexible.

So we have 4 solutions.
The 3 (glfilterapp), and 4 (create a glfilter) are already fully
implemented.
The 1 (copyContext), we have to test it.
The 2, (use the same context) must be used only if 1, 3 and 4 are not
possible (because really a bad way). And it's not yet implemented.

About the solution 4, you will tell me that you can implement a glfilter
because you are using a framework that comes with its own opengl context
management. So the problem is still the same.
(I mean, all the opengl framework focused on draw and esthetic: clutter
etc...)
Then I will say yes, but then I will tell you that the solution is to
implement glwindow with your friendly framework.
I am mean if it's clutter, then you have to create a gstglwindow_clutter.c.
I can help on this.

> Thanks,
> Andres

I hope this can help.

Sincerely

Julien

>
>
> Florent wrote:
> > Hi,
> >
> >
> >> Thanks Jan, I installed gst-plugins-gl from git and did some first
> >> experiments. This seems to be what I'm looking for. Is it correct that
> >> I would use 'glimagesink' and do the drawing using the
> >> client-draw-callback? I'm not experienced with OpenGL, is it safe to
> >> just redraw the part of the scene that shows the video in the
> >> callback?
> >>
> >
> > If i'm not mistaken glimagesink isn't what you are looking for, except
> > if your opengl application supports composite redirection (when
> > glimagesink's textures are redirected inside your own opengl app): the
> > glimagesink will be a dedicated, separate window and gl context. If
> > you need to display gst-gl-originating frames inside your own opengl
> > canvas, then you need to develop some form of a gstreamer sink for
> > your canvas.
> >
> > This is what has been done in both pigment (with pgmimagesink) and
> > clutter (cluttergstsink).
> >
> > The clutter section in gst-gl examples directory shows a
> > gst-gl-originating texture redirection inside a clutter application
> > (using composite redirection).
> >
> >
> >> I'm wondering as this callback function is called by a
> >> gstreamer thread and if the application is doing other OpenGL stuff or
> >> contains more than one video display, multiple threads are doing
> >> OpenGL drawing operations simultaneously.
> >>
> >
> > Your application should take care of the draws sync so that no
> > conflict between opengl operations exists. Also, sharing opengl
> > contexts between apps seems a difficult task.
> >
> >
> >> self.videosink = gst.element_factory_make('glimagesink', 'videosink')
> >> self.videosink.set_property('client-draw-callback', self.draw_callback)
> >>
> >
> > I think the client-draw-callback will offer you a way to manipulate
> > the video texture inside the glimagesink context, not the other way
> > around (drawing video inside your own app).
> >
> > I assume you are only trying to display gst-originating (decoded
> > video) video inside your Gl app. If so, you can prototype things using
> > the fakesink element, by connecting it's handoff signals to a texture
> > upload in your application. See [1] and [2] for more info
> >
> > Florent
> >
> > [1]
> http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-data-spoof.html
> > [2]
> http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-plugins/html/gstreamer-plugins-fakesink.html#GstFakeSink-handoff
> >
> >
> ------------------------------------------------------------------------------
> > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
> > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
> > easily build your RIAs with Flex Builder, the Eclipse(TM)based
> development
> > software that enables intelligent coding and step-through debugging.
> > Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
> > _______________________________________________
> > gstreamer-devel mailing list
> > gstreamer-devel at lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
> >
> >
>
>
>
> ------------------------------------------------------------------------------
> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
> powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
> easily build your RIAs with Flex Builder, the Eclipse(TM)based development
> software that enables intelligent coding and step-through debugging.
> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20090324/ae9ef045/attachment.htm>