Problem with context sharing (glupload) with an OVR thread (Oculus Mobile SDK)

Wed Jan 24 19:35:11 UTC 2018

Hi guys,

I am working on an application that integrates Gstreamer pipeline into
Oculus Mobile SDK (android), with the intention of rendering a video stream
in form of a texture into a VR OpenGL scene.

The test pipeline is simple:
"videotestsrc ! video/x-raw,format=RGBA,width=800,height=600,framerate=1/1 !
glupload ! fakesink sync=1"

Firstly I worked on a application that didn't use the oculus sdk, but a
simple GLSurfaceView. I used context-sharing as in the sdlshare.c gstreamer
example, to be able to pass texture id from appsink/fakesink to the
rendering thread. I go it working, with help of Mattew Watters [1].

The problem is, that when I ported my simple code to the Oculus Mobile SDK
sample (I was using their CubeWorld_SurfaceView example [2]), for some
reason I cannot get it working right. The pipeline seems to be working ok,
it passes me the texture IDs, but these IDs are not recognised by the main
rendering thread (OVR), as if the context sharing never happened... To
exemplify what I mean: I define a few textures in the OVR renderer thread
and their IDs are sth like 2, 6, 7, 8 etc. Gstreamer passes me textures with
ID from 1 until 20 (I buffer up to 20 frames), so actaully the ID from
different threads overlap. During scene rendering if I bind gstreamer ID it
will give me GL_INVALID_OPERATION (so the texture is not present in the
current context) and obviously the texture renders black. If the texture
number (given by the fakesink) by chance hits on of my OVR textures, the OVR
texture is rendered instead of the otherwise black space. It seems clear to
me that the context have separate spaces for gstreamer and the main OVR
thread, even though the context sharing seems to be occurring correctly (no
error/warning etc. in the catlog).
Just to be sure that context sharing works whatsoever, I've created a
separate dummy thread (from the main OVR thread) in which I defined a new
context, sharing the main context. Textures generated in that thread are
indeed visible to the OVR thread... From this I suspect that there is sth
wrong on the gstreamer context-sharing side.

Note: I manage the frame buffer myself trough the fakesink queue, releasing
old frames when a new one is available.

I have attached a context sharing function to the glupload element, called
on the bus callback.

Here I put a few relevant pieces from the catlog, debug level *:6 (the full
log is available here: [3]):

==> Context related calls when gstreamer starts:

https://hastebin.com/adunugemad.swift

==> This happend during starting of the glupload initalzation phase:

https://hastebin.com/ugowasijag.rb

==> Here the bus need-context callback is handled, and my context is passed
to the gluploadelement0

https://hastebin.com/halaqabayo.rb

==> This is how the the rendering (OVR thread) creates the context. Here I
also create my second test thread, that would do what gstreamer does: create
a new, shared context and generate some textures, that I try to access from
my OVR thread. It works like a charm.

https://hastebin.com/piyelexiji.php

==> this is the part in which I manage and retrieve frames from fakesink and
pass to the rendering thread

https://hastebin.com/dovuhowawe.cpp

=================

In this log I don't see any errors or complaint, so I assume the context
sharing was successful. Would I get any error/warning here if share context
was rejected by the element?

When I generate my own textures in a main/side OVR thread I can see:

01-21 13:19:40.442 2214-2292/com.oculus.vrcubeworldsv I/VrCubeWorld:        
new texture created: 7
01-21 13:19:40.494 2214-2292/com.oculus.vrcubeworldsv I/VrCubeWorld:        
new texture created: 8
01-21 13:19:40.509 2214-2292/com.oculus.vrcubeworldsv I/VrCubeWorld:        
new texture created: 9
01-21 13:19:40.511 2214-2292/com.oculus.vrcubeworldsv I/VrCubeWorld:        
new texture created: 10
01-21 13:19:40.513 2214-2292/com.oculus.vrcubeworldsv I/VrCubeWorld:        
new texture created: 11

But the gstreamer will allocate the seme IDs anyhow like here:

01-21 13:19:48.463 2214-2303/com.oculus.vrcubeworldsv V/GStreamer+glmemory:
0:00:08.980197608 0x7a8ef838f0 gstglmemorypbo.c:149:_upload_pbo_memory
upload for texture id:9, with pbo 9 800x600
01-21 13:19:48.463 2214-2303/com.oculus.vrcubeworldsv
V/GStreamer+glbasetexture: 0:00:08.980291801 0x7a8ef838f0
gstglmemory.c:542:gst_gl_memory_texsubimage upload for texture id:9, 800x600
01-21 13:19:48.464 2214-2303/com.oculus.vrcubeworldsv E/GStreamer+gldebug:
0:00:08.980534224 0x7a8ef838f0
gstgldebug.c:303:_gst_gl_debug_callback:<glcontextegl0> high: GL error from
API id:1, Error:glBeginQueryEXT::failed to allocate CPU memory
01-21 13:19:48.464 2214-2303/com.oculus.vrcubeworldsv E/GStreamer+gldebug:
0:00:08.980761532 0x7a8ef838f0
gstgldebug.c:303:_gst_gl_debug_callback:<glcontextegl0> high: GL error from
API id:148, Error:glEndQueryEXT::query name is 0

I can see a GL API error being thrown, but this was also happening in the
working, non-oculus version of the app. So I don't think this is the source
of the problem.

The context sharing function looks like this:

        case GST_MESSAGE_NEED_CONTEXT:
        {

            LOG_INFO("from OpenGL: 0x%08X 0x%08X", *gl_context,
*gl_display);

            // we got gl_display and gl_context and starting of the main
loop

            GstElement *element = GST_ELEMENT (message->src);
            const gchar *name = gst_element_get_name(element);
            //element

            const gchar *context_type;
            gst_message_parse_context_type(message, &context_type);

            LOG_INFO(" Need context callback from '%s' received ",name);
            LOG_INFO(" Context type: %s", context_type);

            // GStreamer vars
            GstGLDisplayEGL *gst_gl_display;
            GstGLContext *gst_gl_context;

            // GStreamer context
            GstContext *context;
            GstStructure *s;

            // get GStreamer display from the native OpenGL EGL vars
            gst_gl_display =
gst_gl_display_egl_new_with_egl_display(gl_display);
            gst_gl_context = gst_gl_context_new_wrapped(gst_gl_display,
*gl_context,
                                                        GST_GL_PLATFORM_EGL,
GST_GL_API_GLES2);

            // set the context
            context = gst_context_new("gst.gl.app_context", TRUE);
            s = gst_context_writable_structure(context);
            gst_structure_set(s, "context", GST_GL_TYPE_CONTEXT,
gst_gl_context, NULL);

            gst_element_set_context(GST_ELEMENT (message->src), context);

            if (context)
                gst_context_unref(context);

        }
            break;

        case GST_MESSAGE_HAVE_CONTEXT: {
            LOG_INFO("Received 'got context' message");
        }

I would be grateful for any help on pinpoint and solving my problem on why
the gstreamer textures are not seen my the rendering thread. The whole issue
is that I do not receive any particular error that I could see in the catlog
or anywhere else, which would help me understand the problem. If more
info/cod sharing is required I'd be more than happy to provide it. 

Thanks,
JW

PS. I use gstreamer prebuild libraries for android, v1.12.0. My target
device is a 64bit arm-v8

Refs
_______________________

[1]
http://gstreamer-devel.966125.n4.nabble.com/Memory-allocation-error-when-trying-to-map-a-video-frame-td4683716.html
[2] https://developer.oculus.com/downloads/package/oculus-mobile-sdk/1.7.0/\
[3] https://uploadfiles.io/mvc2u

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/