OpenGL Texture via GstGLUpload - So close! - Proper Post
Gordon Nickel
gordon.nickel at downstream.com
Thu Oct 1 14:30:19 PDT 2015
Hi Lasse and Matt,
I'm on a very similar path - OpenGL Gst elements sharing a context and textures with Cinder - and I seem to have arrived at a very similar place.
The hastebin links below are broken, is this code anywhere? Or better yet, did you get a full working integration with Cinder?
- Gordon
From: gstreamer-devel [mailto:gstreamer-devel-bounces at lists.freedesktop.org] On Behalf Of Lasse Laursen
Sent: Saturday, January 17, 2015 10:46 AM
To: Discussion of the development of and with GStreamer <gstreamer-devel at lists.freedesktop.org>
Subject: Re: OpenGL Texture via GstGLUpload - So close! - Proper Post
Hey,
Here's an update on the situation:
1) wglGetCurrentContext() returns a non-NULL object, so that seems ok.
2) The wglMakeCurrent call was an overlooked left over from the sdlshare code. It has been commented out.
3) GST_GL_API_OPENGL3 changed to GST_GL_API_OPENGL
4) glshader replaced by glcolorscale
5) Done!
6) Since I don't unref any of the samples/buffers/frames, I'm fairly confident the lack of a mutex lock, should still let textures/data propagate through the system. So for the sake of simplicity, I've not yet implemented this.
I've resorted to using gst_caps_new_simple() just to make sure I'm not messing anything up there. For posterity, here are the current versions of videoTester4.
.H File: http://hastebin.com/ozemufibit.coffee
.CPP File: http://hastebin.com/heyajuwigo.cpp
I may - however - need your help in trying to decode the output that the gstreamer debugger gives me.
For ease of reading, here is the log file at 4 key events.
1) Right after gst_parse_launch() returns: http://hastebin.com/gebunetixa.css
2) Right after changing the state of the pipeline to 5 secs into playback: http://hastebin.com/buxadozudu.md
3) Right at the end of the first call to newVideoSinkSampleCallback(): http://hastebin.com/catafodape.md
4) Right before the texture is about to be inspected by me: http://hastebin.com/lufacoxoco.md
The first thing I looked at is that the Texture ID seems to match fine. As you can see, at the end of the call to newVideoSinkSampleCallback(), a texture with ID 6 is made available (I think?), which is also the texture ID I get from the frame object (and try to render with/inspect). Which leaves me to ponder about the context. It seems as though the gstreamer log is saying that it is creating its own context - which I really shouldn't? Or is it perhaps just misleading debug output?
I'm also a bit curious about the half-sized textures being generated (160x100). I'm also a bit unsure of the Texture Target still. The sdlshare example uses GL_TEXTURE_2D - but even their textures are not sized to the power of 2 - which confuses me a bit to be honest.
I'm keen to hear what you make of the logs?
Regards,
Lasse
On 17-01-2015 13:15, Matthew Waters wrote:
On 17/01/15 22:47, Lasse Laursen wrote:
Hey again,
I can almost taste success. I imagine it being sweet, like sugar. Entirely unlike the bitter lemony sourness of 'not quite there' I have in my mouth now. Help me see the error of my ways!
Inspired by the code Matthew sent my way, I've cobbled together the following object in C++ that is supposed to create a pipeline, share the current OpenGL context with it, grab samples arriving at the appsink and provide the texture ID's to the main thread for usage.
videoTester.h
http://hastebin.com/sutoyuxoci.coffee
videoTester.cpp
http://hastebin.com/gohofababi.cpp
In the main application, the following happens:
Cinder sets up all the win32 app stuff, including an OpenGL context.
VideoTester4 is instantiated with paths pointing to a webm video and a passthrough fragment shader (no parameters).
prepPipeline() is called.
launchPipeline() is called.
The generated Texture ID is inspected via imdebug (an awesome texture debugging tool) via this call:
imdebugTexImage( GL_TEXTURE_2D, mVidTester4->getTexId(), GL_RGBA );
This looks like a solid approach.
A couple of things to try/check.
1. See what the value of gl_context = wglGetCurrentContext() returns. make sure it is not NULL. If it is, make sure you're calling that in the main thread (or wherever your gl context is current).
2. if you call wglMakeCurrent (0, 0) make sure you reactivate it after you're done as cinder might assume it controls the gl context and might not reactivate it by itself.
3. The use of GST_GL_API_OPENGL3 might not match cinder's API usage. Also, the wgl backend does not currently create GL3 contexts yet or contain gl debugging support from the library like the glx backend.
4. In order to remove shader issues, replace glshader by glcolorscale for testing.
5. enable all the gl debugging with GST_DEBUG=gl*:7 and see if the texture handles you get from the callback match up with the textures printed out. It might also mention some things about the context setup in the beginning that might be useful.
6. the lack of locking on mTextureId :)
Note that I intentionally do *not* unref any of the associated frame/sample/buffer objects, just to keep things simple. However, all I seem to get when I use imdebug to inspect the textures, are empty 0,0 sized textures that (not surprisingly) contain nothing at all. I wondered if GL_TEXTURE_2D might be the wrong texture target and have also tried GL_TEXTURE_RECTANGLE without any luck. Is there any place I can read the texture target from?
Anyway, the pipeline does not report any errors along the way, and the gstreamer log only contains a few (harmless?) warnings:
0:00:00.683792474 1836 0CFBEE60 WARN gldisplay gstgldisplay.c:169:gst_gl_display_new: Could not create display. user specified (NULL) (platform: (NULL)), creating dummy
That's normal for win32.
0:00:00.686256033 1836 0CFBEE60 WARN basesrc gstbasesrc.c:3470:gst_base_src_start_complete:<source> pad not activated yet
0:00:00.686752850 1836 0CFBEE60 WARN basesrc gstbasesrc.c:3470:gst_base_src_start_complete:<source> pad not activated yet
These may also be normal.
The texture ID increases as the program continues execution (which is to be expected), and inspecting the Textures in the actual callback thread with imdebug crashes the program. This has to be due to the fact that the callback function executes in a separate thread as Matthew told me last we spoke.
My guess based on this and the above is something simple based on the setup of the gl contexts is off.
I've poked a bit around with the various Gst objects without any luck so far. Any ideas/clues as to what I'm missing or how to narrow down what the problem might be?
Regards,
Lasse
On 16-01-2015 22:21, Matthew Waters wrote:
On 17/01/15 06:42, Lasse Laursen wrote:
Hey Matthew,
Thanks for the reply and especially the example. I'm working on getting it function with the cinder framework. At first it seemed quite odd to hand over the context as a result of a bus message, but I can see the logic in not needing to create a secondary OpenGL context for gstreamer to make textures in, if one already exists. I do have one question though:
Is it vital to turn off the OpenGL context when sharing it with Gstreamer? I've done a little context sharing before and I don't recall having to actively turn any contexts 'off'.
No it is not required to turn off the context.
Regards,
Lasse
On 16-01-2015 00:27, Matthew Waters wrote:
A couple of points added inline.
On 16/01/15 03:14, Lasse Laursen wrote:
Hello!
Apologies for the double post, but my inept handling of Thunderbird caused my last post to be appended to an unrelated thread. Sorry about that Luis dArquer.
For the sake of posterity, here's is the original question:
I feel like I am so close to finally getting through to the other side with this problem, so I'm hoping someone can give me a few clues as to what pieces of the puzzle are missing. I used to use gStreamer as a pure 'decoding' library using this simple pipeline:
uridecodebin uri=" + mPathToVideoFile + " ! videoconvert ! appsink name=sink caps="video/x-raw,format=RGB,pixel-aspect-ratio=1/1";
With this pipeline, I could quite easily grab the frames arriving at the sink and use them however I pleased. Mostly just throwing them on a texture in OpenGL and displaying them various places. This approach is - of course - quite horrible. I'm not making use of gstreamer as a fully-fledged media pipeline. So to rectify that - and because I'd like to run some simple shaders on the video data - I am now using this (almost as) simple pipeline:
uridecodebin uri=" + mPathToVideoFile + " ! glshader name=shader location=" + pathToShader + " vars=\"float silver = float( 0.8 ); \" ! appsink name=sink caps=\"video/x-raw,format=RGB,pixel-aspect-ratio=1/1\"";
1. You should set the appsink caps to contain the memory:GstGLMemory caps features. e.g. video/x-raw(memory:GstGLMemory),format=RGBA... see gst-inspect-1.0 glshader for the supported formats and features. Otherwise, glshader (and any GL element) will download the texture upon seeing the system memory caps.
I make sure to share my current OpenGL context with the glshader element thusly:
mPipeline_ShaderPre = gst_bin_get_by_name( GST_BIN( mGstPipeline ), "shader" );
HGLRC mainContext = ::wglGetCurrentContext();
g_object_set( mPipeline_ShaderPre, "other-context", mainContext, NULL );
2. The "other-context" property expects a GstGLContext rather than a platform specific handle. You can wrap a context handle using gst_gl_context_new_wrapped(). However, in master there is no "other-context" property anymore so the sharing of GL context happens using GstContext. See for example http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/tests/examples/gl/sdl/sdlshare.c#n232 and the GstContext documentation.
Not sure if this is correct to be honest, but I didn't see any errors in gstreamers logs. Anyway - fast forward to my callback function that fires whenever a sample is ready at the sink I proceed to grab a Sample, from which I grab a Buffer, from which I grab a Frame, from which I grab a Texture ID, thusly:
GstSample* gstVideoSinkSample = gst_app_sink_pull_sample( GST_APP_SINK( appsink ) );
GstBuffer* gstBuffer = gst_sample_get_buffer( gstVideoSinkSample );
if ( !gst_video_frame_map( &gstFrame, &gstInfo, gstBuffer, static_cast<GstMapFlags>( GST_MAP_READ | ( GST_MAP_FLAG_LAST << 1 ) ) ) )
{
errorToConsole( true, "Failed to map video frame" );
}
GLuint textureId = *(guint *) gstFrame.data[0];
The result is the number 6. Which I felt was a plausible Texture ID handle. But when I then use a lovely tool named imdebug, and presume that the target for the texture is GL_TEXTURE_2D (which might be wrong), it throws a fit, because it attempts to determine the width and height of said texture via these calls:
glBindTexture(target, texture);
glGetTexLevelParameteriv(target, 0, GL_TEXTURE_WIDTH, &w);
glGetTexLevelParameteriv(target, 0, GL_TEXTURE_HEIGHT, &h);
So clearly something is wrong. Some place along this winding road, I've misunderstood something or overlooked something. I'd much appreciate any suggestions/clues!
I myself am wondering if a callback at the sink is the right place to grab the texture ID, but I'm not sure how to get at it any earlier, and if that's wise. I am also unsure of how long this texture Id is valid if I want to render it on to a quad. I hope someone can shed a bit of light!
It will be valid for as long as it's not overwritten/reused by the producing element which generally means while you keep a ref to the buffer or have the video frame mapped.
Regards,
Lasse
--
Lasse Farnung Laursen
Researcher at the University of Tokyo
www.lasselaursen.com<http://www.lasselaursen.com>
Twitter: @PMP_J<https://twitter.com/PMP_J>
_______________________________________________
gstreamer-devel mailing list
gstreamer-devel at lists.freedesktop.org<mailto:gstreamer-devel at lists.freedesktop.org>
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
--
Lasse Farnung Laursen
Researcher at the University of Tokyo
www.lasselaursen.com<http://www.lasselaursen.com>
Twitter: @PMP_J<https://twitter.com/PMP_J>
_______________________________________________
gstreamer-devel mailing list
gstreamer-devel at lists.freedesktop.org<mailto:gstreamer-devel at lists.freedesktop.org>
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
--
Lasse Farnung Laursen
Researcher at the University of Tokyo
www.lasselaursen.com<http://www.lasselaursen.com>
Twitter: @PMP_J<https://twitter.com/PMP_J>
_______________________________________________
gstreamer-devel mailing list
gstreamer-devel at lists.freedesktop.org<mailto:gstreamer-devel at lists.freedesktop.org>
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
--
Lasse Farnung Laursen
Researcher at the University of Tokyo
www.lasselaursen.com<http://www.lasselaursen.com>
Twitter: @PMP_J<https://twitter.com/PMP_J>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20151001/0d742f09/attachment-0001.html>
More information about the gstreamer-devel
mailing list