OpenGL Texture via GstGLUpload - So close! - Proper Post
Lasse Laursen
lasse at lasselaursen.com
Sat Jan 17 10:46:21 PST 2015
Hey,
Here's an update on the situation:
1) wglGetCurrentContext() returns a non-NULL object, so that seems ok.
2) The wglMakeCurrent call was an overlooked left over from the sdlshare
code. It has been commented out.
3) GST_GL_API_OPENGL3 changed to GST_GL_API_OPENGL
4) glshader replaced by glcolorscale
5) Done!
6) Since I don't unref any of the samples/buffers/frames, I'm fairly
confident the lack of a mutex lock, should still let textures/data
propagate through the system. So for the sake of simplicity, I've not
yet implemented this.
I've resorted to using gst_caps_new_simple() just to make sure I'm not
messing anything up there. For posterity, here are the current versions
of videoTester4.
.H File: http://hastebin.com/ozemufibit.coffee
.CPP File: http://hastebin.com/heyajuwigo.cpp
I may - however - need your help in trying to decode the output that the
gstreamer debugger gives me.
For ease of reading, here is the log file at 4 key events.
1) Right after gst_parse_launch() returns:
http://hastebin.com/gebunetixa.css
2) Right after changing the state of the pipeline to 5 secs into
playback: http://hastebin.com/buxadozudu.md
3) Right at the end of the first call to newVideoSinkSampleCallback():
http://hastebin.com/catafodape.md
4) Right before the texture is about to be inspected by me:
http://hastebin.com/lufacoxoco.md
The first thing I looked at is that the Texture ID seems to match fine.
As you can see, at the end of the call to newVideoSinkSampleCallback(),
a texture with ID 6 is made available (I think?), which is also the
texture ID I get from the frame object (and try to render with/inspect).
Which leaves me to ponder about the context. It seems as though the
gstreamer log is saying that it is creating its own context - which I
really shouldn't? Or is it perhaps just misleading debug output?
I'm also a bit curious about the half-sized textures being generated
(160x100). I'm also a bit unsure of the Texture Target still. The
sdlshare example uses GL_TEXTURE_2D - but even their textures are not
sized to the power of 2 - which confuses me a bit to be honest.
I'm keen to hear what you make of the logs?
Regards,
Lasse
On 17-01-2015 13:15, Matthew Waters wrote:
> On 17/01/15 22:47, Lasse Laursen wrote:
>> Hey again,
>>
>> I can almost taste success. I imagine it being sweet, like sugar.
>> Entirely unlike the bitter lemony sourness of 'not quite there' I
>> have in my mouth now. Help me see the error of my ways!
>>
>> Inspired by the code Matthew sent my way, I've cobbled together the
>> following object in C++ that is supposed to create a pipeline, share
>> the current OpenGL context with it, grab samples arriving at the
>> appsink and provide the texture ID's to the main thread for usage.
>>
>> videoTester.h
>> http://hastebin.com/sutoyuxoci.coffee
>>
>> videoTester.cpp
>> http://hastebin.com/gohofababi.cpp
>>
>> In the main application, the following happens:
>>
>> Cinder sets up all the win32 app stuff, including an OpenGL context.
>> VideoTester4 is instantiated with paths pointing to a webm video and
>> a passthrough fragment shader (no parameters).
>> prepPipeline() is called.
>> launchPipeline() is called.
>> The generated Texture ID is inspected via imdebug (an awesome texture
>> debugging tool) via this call:
>> imdebugTexImage( GL_TEXTURE_2D, mVidTester4->getTexId(), GL_RGBA );
>
> This looks like a solid approach.
>
> A couple of things to try/check.
>
> 1. See what the value of gl_context = wglGetCurrentContext() returns.
> make sure it is not NULL. If it is, make sure you're calling that in
> the main thread (or wherever your gl context is current).
> 2. if you call wglMakeCurrent (0, 0) make sure you reactivate it after
> you're done as cinder might assume it controls the gl context and
> might not reactivate it by itself.
> 3. The use of GST_GL_API_OPENGL3 might not match cinder's API usage.
> Also, the wgl backend does not currently create GL3 contexts yet or
> contain gl debugging support from the library like the glx backend.
> 4. In order to remove shader issues, replace glshader by glcolorscale
> for testing.
> 5. enable all the gl debugging with GST_DEBUG=gl*:7 and see if the
> texture handles you get from the callback match up with the textures
> printed out. It might also mention some things about the context
> setup in the beginning that might be useful.
> 6. the lack of locking on mTextureId :)
>
>> Note that I intentionally do *not* unref any of the associated
>> frame/sample/buffer objects, just to keep things simple. However, all
>> I seem to get when I use imdebug to inspect the textures, are empty
>> 0,0 sized textures that (not surprisingly) contain nothing at all. I
>> wondered if GL_TEXTURE_2D might be the wrong texture target and have
>> also tried GL_TEXTURE_RECTANGLE without any luck. Is there any place
>> I can read the texture target from?
>>
>> Anyway, the pipeline does not report any errors along the way, and
>> the gstreamer log only contains a few (harmless?) warnings:
>>
>> 0:00:00.683792474 1836 0CFBEE60 WARN gldisplay
>> gstgldisplay.c:169:gst_gl_display_new: Could not create display. user
>> specified (NULL) (platform: (NULL)), creating dummy
>
> That's normal for win32.
>
>> 0:00:00.686256033 1836 0CFBEE60 WARN basesrc
>> gstbasesrc.c:3470:gst_base_src_start_complete:<source> pad not
>> activated yet
>> 0:00:00.686752850 1836 0CFBEE60 WARN basesrc
>> gstbasesrc.c:3470:gst_base_src_start_complete:<source> pad not
>> activated yet
>
> These may also be normal.
>
>> The texture ID increases as the program continues execution (which is
>> to be expected), and inspecting the Textures in the actual callback
>> thread with imdebug crashes the program. This has to be due to the
>> fact that the callback function executes in a separate thread as
>> Matthew told me last we spoke.
>
> My guess based on this and the above is something simple based on the
> setup of the gl contexts is off.
>
>> I've poked a bit around with the various Gst objects without any luck
>> so far. Any ideas/clues as to what I'm missing or how to narrow down
>> what the problem might be?
>>
>> Regards,
>> Lasse
>>
>> On 16-01-2015 22:21, Matthew Waters wrote:
>>>
>>> On 17/01/15 06:42, Lasse Laursen wrote:
>>>> Hey Matthew,
>>>>
>>>> Thanks for the reply and especially the example. I'm working on
>>>> getting it function with the cinder framework. At first it seemed
>>>> quite odd to hand over the context as a result of a bus message,
>>>> but I can see the logic in not needing to create a secondary OpenGL
>>>> context for gstreamer to make textures in, if one already exists. I
>>>> do have one question though:
>>>>
>>>> Is it vital to turn off the OpenGL context when sharing it with
>>>> Gstreamer? I've done a little context sharing before and I don't
>>>> recall having to actively turn any contexts 'off'.
>>>
>>> No it is not required to turn off the context.
>>>
>>>> Regards,
>>>> Lasse
>>>>
>>>> On 16-01-2015 00:27, Matthew Waters wrote:
>>>>> A couple of points added inline.
>>>>>
>>>>> On 16/01/15 03:14, Lasse Laursen wrote:
>>>>>> Hello!
>>>>>>
>>>>>> Apologies for the double post, but my inept handling of
>>>>>> Thunderbird caused my last post to be appended to an unrelated
>>>>>> thread. Sorry about that Luis dArquer.
>>>>>>
>>>>>> For the sake of posterity, here's is the original question:
>>>>>>
>>>>>> I feel like I am so close to finally getting through to the other
>>>>>> side with this problem, so I'm hoping someone can give me a few
>>>>>> clues as to what pieces of the puzzle are missing. I used to use
>>>>>> gStreamer as a pure 'decoding' library using this simple pipeline:
>>>>>>
>>>>>> uridecodebin uri=" + mPathToVideoFile + " ! videoconvert !
>>>>>> appsink name=sink
>>>>>> caps="video/x-raw,format=RGB,pixel-aspect-ratio=1/1";
>>>>>>
>>>>>> With this pipeline, I could quite easily grab the frames arriving
>>>>>> at the sink and use them however I pleased. Mostly just throwing
>>>>>> them on a texture in OpenGL and displaying them various places.
>>>>>> This approach is - of course - quite horrible. I'm not making use
>>>>>> of gstreamer as a fully-fledged media pipeline. So to rectify
>>>>>> that - and because I'd like to run some simple shaders on the
>>>>>> video data - I am now using this (almost as) simple pipeline:
>>>>>>
>>>>>> uridecodebin uri=" + mPathToVideoFile + " ! glshader name=shader
>>>>>> location=" + pathToShader + " vars=\"float silver = float( 0.8 );
>>>>>> \" ! appsink name=sink
>>>>>> caps=\"video/x-raw,format=RGB,pixel-aspect-ratio=1/1\"";
>>>>>
>>>>> 1. You should set the appsink caps to contain the
>>>>> memory:GstGLMemory caps features. e.g.
>>>>> video/x-raw(memory:GstGLMemory),format=RGBA... see gst-inspect-1.0
>>>>> glshader for the supported formats and features. Otherwise,
>>>>> glshader (and any GL element) will download the texture upon
>>>>> seeing the system memory caps.
>>>>>
>>>>>> I make sure to share my current OpenGL context with the glshader
>>>>>> element thusly:
>>>>>>
>>>>>> mPipeline_ShaderPre = gst_bin_get_by_name( GST_BIN( mGstPipeline
>>>>>> ), "shader" );
>>>>>> HGLRC mainContext = ::wglGetCurrentContext();
>>>>>> g_object_set( mPipeline_ShaderPre, "other-context", mainContext,
>>>>>> NULL );
>>>>>
>>>>> 2. The "other-context" property expects a GstGLContext rather than
>>>>> a platform specific handle. You can wrap a context handle using
>>>>> gst_gl_context_new_wrapped(). However, in master there is no
>>>>> "other-context" property anymore so the sharing of GL context
>>>>> happens using GstContext. See for example
>>>>> http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/tests/examples/gl/sdl/sdlshare.c#n232
>>>>> and the GstContext documentation.
>>>>>
>>>>>> Not sure if this is correct to be honest, but I didn't see any
>>>>>> errors in gstreamers logs. Anyway - fast forward to my callback
>>>>>> function that fires whenever a sample is ready at the sink I
>>>>>> proceed to grab a Sample, from which I grab a Buffer, from which
>>>>>> I grab a Frame, from which I grab a Texture ID, thusly:
>>>>>>
>>>>>> GstSample* gstVideoSinkSample = gst_app_sink_pull_sample(
>>>>>> GST_APP_SINK( appsink ) );
>>>>>> GstBuffer* gstBuffer = gst_sample_get_buffer( gstVideoSinkSample );
>>>>>> if ( !gst_video_frame_map( &gstFrame, &gstInfo, gstBuffer,
>>>>>> static_cast<GstMapFlags>( GST_MAP_READ | ( GST_MAP_FLAG_LAST << 1
>>>>>> ) ) ) )
>>>>>> {
>>>>>> errorToConsole( true, "Failed to map video frame" );
>>>>>> }
>>>>>> GLuint textureId = *(guint *) gstFrame.data[0];
>>>>>>
>>>>>> The result is the number 6. Which I felt was a plausible Texture
>>>>>> ID handle. But when I then use a lovely tool named imdebug, and
>>>>>> presume that the target for the texture is GL_TEXTURE_2D (which
>>>>>> might be wrong), it throws a fit, because it attempts to
>>>>>> determine the width and height of said texture via these calls:
>>>>>>
>>>>>> glBindTexture(target, texture);
>>>>>> glGetTexLevelParameteriv(target, 0, GL_TEXTURE_WIDTH, &w);
>>>>>> glGetTexLevelParameteriv(target, 0, GL_TEXTURE_HEIGHT, &h);
>>>>>>
>>>>>> So clearly something is wrong. Some place along this winding
>>>>>> road, I've misunderstood something or overlooked something. I'd
>>>>>> much appreciate any suggestions/clues!
>>>>>>
>>>>>> I myself am wondering if a callback at the sink is the right
>>>>>> place to grab the texture ID, but I'm not sure how to get at it
>>>>>> any earlier, and if that's wise. I am also unsure of how long
>>>>>> this texture Id is valid if I want to render it on to a quad. I
>>>>>> hope someone can shed a bit of light!
>>>>>
>>>>> It will be valid for as long as it's not overwritten/reused by the
>>>>> producing element which generally means while you keep a ref to
>>>>> the buffer or have the video frame mapped.
>>>>>
>>>>>> Regards,
>>>>>> Lasse
>>>>>>
>>>>>> --
>>>>>> Lasse Farnung Laursen
>>>>>> Researcher at the University of Tokyo
>>>>>> www.lasselaursen.com <http://www.lasselaursen.com>
>>>>>> Twitter: @PMP_J <https://twitter.com/PMP_J>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> gstreamer-devel mailing list
>>>>> gstreamer-devel at lists.freedesktop.org
>>>>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>>>
>>>> --
>>>> Lasse Farnung Laursen
>>>> Researcher at the University of Tokyo
>>>> www.lasselaursen.com <http://www.lasselaursen.com>
>>>> Twitter: @PMP_J <https://twitter.com/PMP_J>
>>>
>>>
>>>
>>> _______________________________________________
>>> gstreamer-devel mailing list
>>> gstreamer-devel at lists.freedesktop.org
>>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>> --
>> Lasse Farnung Laursen
>> Researcher at the University of Tokyo
>> www.lasselaursen.com <http://www.lasselaursen.com>
>> Twitter: @PMP_J <https://twitter.com/PMP_J>
>
>
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
--
Lasse Farnung Laursen
Researcher at the University of Tokyo
www.lasselaursen.com <http://www.lasselaursen.com>
Twitter: @PMP_J <https://twitter.com/PMP_J>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20150117/64ab60eb/attachment-0001.html>
More information about the gstreamer-devel
mailing list