some notes for opengl plugin
comicfans44
comicfans44 at gmail.com
Wed Aug 27 17:47:20 PDT 2014
recently I'm learning intergrating gstreamer into my application (using
opengl plugin and context share),gstreamer and its opengl plugin is
powerful,
but the opengl plugin lacks some "key notes " in document which is
important
for context-share at some point. you can only find them in the demo/src.
without this, some hard-to-debug problem may block others who try to
integrate gstreamer opengl. so I wrote them down here, wish these notes
going
into document and help others to use opengl plugin easiler.
feel free to correct .
following step shows the way I integrate gstreamer with opengl
context-share
into my app (which comes from qglwtextureshare demo in gst-plugins-bad
tests/example/gl/qt):
app create it's opengl context (ref as main context)->
gstreamer create share context with main context(ref as gstreamer's
context)->
use fakesink hands-off signal to receive GstBuffer wrapped texture back ->
draw texture on main context
in this way, gstreamer process images in another thread opengl,you receive
the result texture ,and do anything you want in your main context.
note 1: you must unbind your main context until gstreamer's context create
complete,(that is, gst_gl_context_create function returned) I
haven't
find too much about this restriction, but there's notes at
qglrenderer.cpp:73
for nvidia card (windows), not to do this will lead gstreamer's context
create code failed everytime. (you may GetLastError=0xc00710dd
see
http://stackoverflow.com/questions/21074447/undocumented-error-during-opengl-context-creation)
I didn't test the incorrect code under Linux,but I guess result will be
same
for ATI card (Linux/windows), not do this will lead no error code
reported
when gstreamer creating context, but main context window just black!
you may find sometimes after running some other correct context-share
app , the incorrect code magically worked !(I'm not joking) , you
may
think your app "worked" ,but if you restart PC , everything goes
crazy.
for some toolkit(likes Qt) ,opengl context is created on GUI thread, and
you
can only ensure this context binded to current thread in GUI draw callback.
you can share main context to gstreamer as follows:
1.1. get main context in the GUI draw callback (or any position you're sure
main context is binded to current thread.like
Qt::QGLWidget::initializeGL
use wglGetCurrentContext or something equal)
1.2. unbind it (QGLContext::doneCurrent or something equal), you can
reference
qglrenderer.cpp:64
1.3 passing it to gstreamer-gl element's other-context property
(reference to qglrenderer.cpp:64). you just need to do this for only
one opengl element. currently I set it to last opengl element in
pipeline
and it works OK.
1.4 blocking wait in GUI draw callback until gstreamer's context share code
complete,
(or you must ensure GUI-toolkit will not rebind the previously unbinded
main context back or call opengl function after draw callback returned)
.
then you can rebind your main context back and do anything as usual.
since there's no "context-share complete" signal from gstreamer , so you
don't know which timepoint your blocking wait can wakeup, one choice is
waiting until whole pipeline enter PAUSED state , like the qglwtextureshare
demo. I'm not sure the state change workflow inside gstreamer .seems this
only
works when src supports poll mode (like videotestsrc/filesrc ) ,
but if you're using appsrc in push mode , PAUSED state is not enough,
you must wait pipeline enter PLAYING state (wait the gl element which you
set
other-context enter PLAYING state not helped ). if your data is not ready
at the point, whole pipeline will not enter PLAYING state, you can pushing
"fake data" once to appsrc at beginning (before waiting ) to avoid the UI
block.
you should arrange app logic carefully , if you block waiting in UI thread
and
use main-loop based timer (g_idle_add or QTimer) in UI thread to trigger
data push, it may dead-lock (main loop blocked by waiting , but pipeline
waiting main loop timer callback to push)
if you are using fakesink/appsink to receive data back, you can also wakeup
UI waiting when first buffer arrvied.
so here comes the point I think opengl plugin can be improved, add a
"context share complete" signal helps a lot, or maybe some other
ways to create share-context immediately, not depends on state change.
note 2. when using the GstBuffer wrapped texture in main context, you
may flush the gstreamer context first. opengl may cache draw commands,
so the gstreamer's texture may not be complete before being used in main
context.
for nvidia card, the gstreamer's texture is always rendered completely
in main context use(seems nv driver control this)
for ATI card,without this, main context can render some
gstreamer's textures correctly at first, but may fail after a while
or
elements dynamically changed. you will find many random garbage
or previous frame drawed on main context.
you may reference qglrenderer.cpp:122 to get code hint.(peek GLMemory
from result GstBuffer, add flush callback to GLMemory's context by
gst_gl_context_thread_add, it will execute flush callback in
gstreamer's opengl thread, wait it complete and return)
qglwtextureshare demo cached 3 GstBuffers before drawing on main context
(pipeline.cpp:178), reduce this number to 1 ,you will notice flick in
app without flushing .
That's all, hopes it helped.
More information about the gstreamer-devel
mailing list