Memory allocation error when trying to map a video frame

[Matthew Waters]

On 10/07/17 00:22, jeremi.wojcicki wrote:
> Hi Matt,
>
> Thanks for helping me out with it. Here's what I have done so far:
>
> 1. I reported the bug as asked. I have changed my pipeline to generate test
> source already in GPU memory:
>
> gltestsrc is-live=true !
> video/x-raw(memory:GLMemory),format=RGBA,width=1280,height=720,framerate=11/3,pixel-aspect-ratio=1/1
> ! glcolorconvert ! video/x-raw(memory:GLMemory),format=RGB ! appsink
>
> Then the error does not occur, and I believe it is more efficient (less
> memory copying etc.).

Right, the errors would only appear when you are transferring data
to/from RAM/GPU as that's the only place that uses GL queries.

> 2. Regarding undefined references - I have created a temporary, primitive
> yet working solution: a circular buffer where I store my frames. As you can
> see here  https://hastebin.com/ucojaqunak.cpp
> <https://hastebin.com/ucojaqunak.cpp>   I unmap the old frames at the
> beginning of the function and then overwrite it with a new one. I intend to
> write a more elaborate mechanism of sharing it between threads in the
> future, but I guess its good enough for the testing the rest. I check my
> textures in the drawing thread with glIsTexture and they seem to be all
> valid. I also can seen that the textureID value is increasing by one with
> each frame and then jumps back to the starting value (2 or sth), which would
> indicate that the frames stay in memory as desired.

They stay in memory and are reused through the GstBufferPool mechanism. 
However the usage of them is not synchonized and data races may occur as
you are not holding on to the reference to the OpenGL texture and as
such, GStreamer may write data into it while you're reading data from
it.  Keeping a reference on the buffer/sample/memory while you're using
the texture fixes this.

> 3. The synchronization and randomly occurring black texture, is the most
> difficult topic, that I do not fully understand yet. Let me describe the
> problem a bit more in detail. Whether the texture will be black or not is
> somewhat "decided" on application startup, there is not flickering or
> anything of the image during execution. If the texture will show up video
> keeps on going flawlessly. If it will not show up in the beginning it will
> stay like this, so you have to restart. I had some gut feeling that it may
> be related to some synchronization issues but I didn't have a clear idea how
> to approach the problem.
>
> You wrote that while using multiple OpenGL contexts I am supposed to ensure
> proper synchronization. This somewhat already confuses me, because I though
> that after sharing my OpenGL renderers context with GST elements there
> aren't multiple contexts anymore, but multiple threads instead. How is it
> then? (excuse my noob questions, as I am not experienced in gstreamer nor
> opengl that well).
>
> Let me understand a bit better the issue of synchronization between opengl
> and gstreamer. I shared OpenGL context between several gstreamer elements
> (in my case all of them: source, glcolorconvert and appsink). Do these
> elements already manage the synchronization between each other or should I
> take care of it myself, since I "forced" them to share the context?

It's not like that at all.

The OpenGL context you pass into GStreamer using the GstContext query
(aka, application GL context), is not touched by GStreamer at all as it
has it's own internal OpenGL context (aka GStreamer GL context). 
GStreamer will create it's GL context to be shared with the application
GL context so that some GL resources (textures, shaders, etc) can be
shared between them.  However the GL state machine of each GL context is
completely separate.

When OpenGL context's are shared, there is by definition, more than one
OpenGL context in the application.  As a result, synchronizing these
context's is required when accessing data that is used by more than one
OpenGL context.

> I am guessing that the most critical part is to sync my rendering thread
> with the gst pipeline. Can it mess up my own rendering pipeline if I don't?
> (in the sense, that when my thread is performing some drawing operation to
> the screen buffer the gst will perform simultaneously some other operations
> to a framebuffer that can cause some unexpected behaviors in both of them)?

No, GStreamer's and the application's GL contexts are mostly separate
and have separate GL state attached to them.

> You proposed to use GstGLSyncMeta to ensure the synchronization. I could not
> find any exhaustive documentation on the topic, so I would kindly ask you to
> guide me through the process.

Look up any references to glFenceSync(), as that's what GstGLSyncMeta
uses internally.

gst_gl_sync_meta_set_sync_point() inserts an event into the GL command
stream that can be waited/polled on later.  This is performed by all
upstream OpenGL elements in the pipeline at the end of their OpenGL
processing.

As a consumer, one should call gst_gl_sync_meta_wait_gpu/cpu() when you
need access to the data and depending on if your going to access from
RAM or from the GPU with more OpenGL commands.

> So lets assume I would like to set the syncpoint in my opengl rendering
> thread and make the appsink wait. Should it be sth like this?
>
> // the renderer thread
> OnDraw(){
>
> // ... some drawing calls etc.
>
> glFinish();
> gst_gl_sync_meta_set_sync_point (sync_meta,  context);

These two calls should only be gst_gl_sync_meta_wait_gpu ().

> }
>
> Should sync_meta be taken once from gst at startup or updated continuously
> during run time?
> On the gstreamer side should I add the gst_gl_sync_meta_wait in the appsink
> callback?

Where you call gst_gl_sync_meta_wait() is up to you however, there must
be a OpenGL context current in the thread in which you call
gst_gl_sync_meta_wait_*().

> Will it work if appsink frames arrive less often (like 15fps) then the
> opengl scene draws (~60fps)?

Yes, as long as you keep references to your data correctly.

> Sorry for an avalanche of questions, but without some help I like a dog in
> the fog :)
>
> Thanks,
> Jeremi
>
>
>
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Memory-allocation-error-when-trying-to-map-a-video-frame-tp4683716p4683744.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
>


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 516 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20170710/5c8a9675/attachment.sig>