Vaapidecoder + appsink into eglImage

[Davide Molteni]

Thanks Victor,
We are using gstreamer 1.16 so we don't have access to va components yet.

So I guess I have to use the glupload component before the appsink in my decoders pipelines. 
Is there a sample I can refer to in order to handle the texture in the appsink callback?

Thanks,
Davide  

-----Original Message-----
From: gstreamer-devel <gstreamer-devel-bounces at lists.freedesktop.org> On Behalf Of Victor Jaquez via gstreamer-devel
Sent: Wednesday, July 19, 2023 11:22 AM
To: gstreamer-devel at lists.freedesktop.org
Cc: Victor Jaquez <vjaquez at igalia.com>
Subject: Re: Vaapidecoder + appsink into eglImage

On Tue, 18 Jul 2023 at 16:07, Davide Molteni via gstreamer-devel wrote:
> Hi all,
>
> I'm using a gstreamer VAAPI decoder pipeline with an appsink and I'm 
> binding the samples on an egl texture where data is the pointer to 
> GstMapInfo->data from gst_buffer_map. Eglsurface is then rendered on 
> framebuffers, since I'm running on an embedded platform without window manger.
> 
> Pseudocode of my appsink looks like the following:
> 
> GstBuffer* buffer = gst_sample_get_buffer(sample); GstMapInfo 
> memoryInfo; gst_buffer_map(buffer, &memoryInfo, GST_MAP_READ);
> 
> /*opengGL stuff*/
> glTexImage2D(GL_TEXTURE_2D, 0, GL_LUMINANCE, width, height, 0, 
> GL_LUMINANCE, GL_UNSIGNED_BYTE, memoryInfo.data);
> 
> gst_buffer_unmap(buffer, &memoryInfo);
> 
> 
> The current code seems to work correctly in both cases when I set the 
> caps of the decoder to video/x-raw and when I set it to 
> video/x-raw(memory:VASurface), which is kind of strange, I would 
> expect a crash in case of VASurface, am I missing something here?

The buffers pushed with caps features memory:VASurface and memory:System
(default) are the same. The caps features are just to prioritize VASurface at negotiation.

> 
> Anyway I'm looking for the correct way to create the eglImage from the 
> VASurface, something that could look like this:

Why would you do this manually when you have glupload element that do that for you? You would get a texture with a EGLImage.

> 
>        VADRMPRIMESurfaceDescriptor va_surface_descriptor;
>        vaExportSurfaceHandle(va_display, va_surface, 
> VA_SURFACE_ATTRIB_MEM_TYPE_DRM_PRIME_2, VA_EXPORT_SURFACE_READ_ONLY, 
> &va_surface_descriptor);
> 
>        EGLAttrib egl_img_attributes[] = {
>               EGL_LINUX_DRM_FOURCC_EXT, va_surface_descriptor.layers[0].drm_format,
>               EGL_WIDTH, va_surface_descriptor.width,
>               EGL_HEIGHT, va_surface_descriptor.height,
>               EGL_DMA_BUF_PLANE0_FD_EXT, va_surface_descriptor.objects[va_surface_descriptor.layers[0].object_index[0]].fd,
>               EGL_DMA_BUF_PLANE0_OFFSET_EXT, va_surface_descriptor.layers[0].offset[0],
>               EGL_DMA_BUF_PLANE0_PITCH_EXT, va_surface_descriptor.layers[0].pitch[0],
>               EGL_NONE
>        };
>        hEglImage = eglCreateImage(eglGetCurrentDisplay(), EGL_NO_CONTEXT, EGL_LINUX_DMA_BUF_EXT, NULL, egl_img_attributes);
>               glEGLImageTargetTexture2DOES(GL_TEXTURE_2D, hEglImage);
> 
> But I need to get an handle to va_display (VADisplay) and va_image 
> (which is the VASurfaceID) from the sample pulled from the appsink.
> 
> Is it possible to access them from the Buffer Metadata? Probably doing 
> something like this in the appsink callback:
> 
> GstBuffer* buffer = gst_sample_get_buffer(sample);
> GstVaapiVideoMeta* const meta = 
> gst_buffer_get_vaapi_video_meta(buffer);
> Is it the correct approach?

All that was possible through a very heavy library and API, which turned to be impossible to maintain, so it was buried down in the plugin. You can look at the code and extract the structures, either in the buffer or in its meta, and go deep in them until you get these two values.

Doable but fragile because there aren't ABI promises.

Another alternative is to move to gstva plugins which provides, in my opinion, a better developer story.

You can check a similar use case as yours, using gstva, in this example:

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-bad/tests/examples/va/main.c

In this example X11 is used, but you can change it easily to EGL.

vmjl

> 
> Thanks,
> Davide