Vaapidecoder + appsink into eglImage

[Victor Jaquez]

On Tue, 18 Jul 2023 at 16:07, Davide Molteni via gstreamer-devel wrote:
> Hi all,
>
> I'm using a gstreamer VAAPI decoder pipeline with an appsink and I'm binding
> the samples on an egl texture where data is the pointer to GstMapInfo->data
> from gst_buffer_map. Eglsurface is then rendered on framebuffers, since I'm
> running on an embedded platform without window manger.
> 
> Pseudocode of my appsink looks like the following:
> 
> GstBuffer* buffer = gst_sample_get_buffer(sample);
> GstMapInfo memoryInfo;
> gst_buffer_map(buffer, &memoryInfo, GST_MAP_READ);
> 
> /*opengGL stuff*/
> glTexImage2D(GL_TEXTURE_2D, 0, GL_LUMINANCE, width, height, 0, GL_LUMINANCE, GL_UNSIGNED_BYTE, memoryInfo.data);
> 
> gst_buffer_unmap(buffer, &memoryInfo);
> 
> 
> The current code seems to work correctly in both cases when I set the caps of
> the decoder to video/x-raw and when I set it to video/x-raw(memory:VASurface),
> which is kind of strange, I would expect a crash in case of VASurface, am I
> missing something here?

The buffers pushed with caps features memory:VASurface and memory:System
(default) are the same. The caps features are just to prioritize VASurface at
negotiation.

> 
> Anyway I'm looking for the correct way to create the eglImage from the
> VASurface, something that could look like this:

Why would you do this manually when you have glupload element that do that for
you? You would get a texture with a EGLImage.

> 
>        VADRMPRIMESurfaceDescriptor va_surface_descriptor;
>        vaExportSurfaceHandle(va_display, va_surface, VA_SURFACE_ATTRIB_MEM_TYPE_DRM_PRIME_2, VA_EXPORT_SURFACE_READ_ONLY, &va_surface_descriptor);
> 
>        EGLAttrib egl_img_attributes[] = {
>               EGL_LINUX_DRM_FOURCC_EXT, va_surface_descriptor.layers[0].drm_format,
>               EGL_WIDTH, va_surface_descriptor.width,
>               EGL_HEIGHT, va_surface_descriptor.height,
>               EGL_DMA_BUF_PLANE0_FD_EXT, va_surface_descriptor.objects[va_surface_descriptor.layers[0].object_index[0]].fd,
>               EGL_DMA_BUF_PLANE0_OFFSET_EXT, va_surface_descriptor.layers[0].offset[0],
>               EGL_DMA_BUF_PLANE0_PITCH_EXT, va_surface_descriptor.layers[0].pitch[0],
>               EGL_NONE
>        };
>        hEglImage = eglCreateImage(eglGetCurrentDisplay(), EGL_NO_CONTEXT, EGL_LINUX_DMA_BUF_EXT, NULL, egl_img_attributes);
>               glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, hEglImage);
> 
> But I need to get an handle to va_display (VADisplay) and va_image (which is
> the VASurfaceID) from the sample pulled from the appsink.
> 
> Is it possible to access them from the Buffer Metadata? Probably doing
> something like this in the appsink callback:
> 
> GstBuffer* buffer = gst_sample_get_buffer(sample);
> GstVaapiVideoMeta* const meta = gst_buffer_get_vaapi_video_meta(buffer);
> Is it the correct approach?

All that was possible through a very heavy library and API, which turned to be
impossible to maintain, so it was buried down in the plugin. You can look at the
code and extract the structures, either in the buffer or in its meta, and go
deep in them until you get these two values.

Doable but fragile because there aren't ABI promises.

Another alternative is to move to gstva plugins which provides, in my opinion, a
better developer story.

You can check a similar use case as yours, using gstva, in this example:

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-bad/tests/examples/va/main.c

In this example X11 is used, but you can change it easily to EGL.

vmjl

> 
> Thanks,
> Davide