How to consume VAAPI VASurface in appsink

Thu Aug 9 19:34:10 UTC 2018

Is this a recent addition? In my plugin, from gst-inspect (see below), I see that I have version 1.8.3 of the plugin, and the src sink only supports video/x-raw(memory:VASurface), video/x-raw(meta:GstVideoGLTextureUploadMeta), and video/x-raw. 

I'm also wondering how to find out the caps beyond "video/x-raw". That's what gst_structure_get_name() tells me. However, I don't know how to get to the part between the parentheses ("memory:VASurface" or "meta:GstVideoGLTextureUploadMeta").

Thanks for the reference to glupload; I'll have a look. 

Sorry for the rookie questions, I'm definitely new to gstreamer & libva internals. 

Plugin Details:
  Name                     vaapi
  Description              VA-API based elements
  Filename                 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvaapi.so
  Version                  1.8.3
  License                  LGPL
  Source module            gstreamer-vaapi
  Source release date      2016-06-09
  Binary package           gstreamer-vaapi
  Origin URL               http://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer

[...]

SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:VASurface)
                 format: { ENCODED, I420, YV12, NV12 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
         interlace-mode: progressive
      video/x-raw(meta:GstVideoGLTextureUploadMeta)
                 format: { RGBA, BGRA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw
                 format: { I420, YV12, YUY2, UYVY, AYUV, RGBx, BGRx, xRGB, xBGR, RGBA, BGRA, ARGB, ABGR, RGB, BGR, Y41B, Y42B, YVYU, Y444, v210, v216, NV12, NV21, NV16, NV61, NV24, GRAY8, GRAY16_BE, GRAY16_LE, v308,
RGB16, BGR16, RGB15, BGR15, UYVP, A420, RGB8P, YUV9, YVU9, IYU1, ARGB64, AYUV64, r210, I420_10LE, I420_10BE, I422_10LE, I422_10BE, Y444_10LE, Y444_10BE, GBR, GBR_10LE, GBR_10BE, NV12_64Z32, A420_10LE, A420_10BE, A422
_10LE, A422_10BE, A444_10LE, A444_10BE }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
         interlace-mode: progressive

-----Original Message-----
From: gstreamer-devel <gstreamer-devel-bounces at lists.freedesktop.org> On Behalf Of Víctor Jáquez
Sent: Thursday, August 09, 2018 9:47 AM
To: Discussion of the development of and with GStreamer <gstreamer-devel at lists.freedesktop.org>
Subject: Re: How to consume VAAPI VASurface in appsink

On Thu, 09 Aug 2018 at 15:56, Galoppo, Nico wrote:
> Thanks!
> 
> However, the vaapipostproc doesn't seem to support 
> "video/x-raw(memory:DMABuf)" on its source pad. I tried it out as follows:
> 
> gst-launch-1.0 -v souphttpsrc
> location=https://www.freedesktop.org/software/gstreamer-sdk/data/media
> /sintel_trailer-480p.webm ! matroskademux ! vaapidecodebin ! 
> vaapipostproc !
> "video/x-raw(memory:DMABuf)" ! fakesink

At least the pipeline works for me. Btw, you could remove the vaapipostproc since vaapidecodebin already contains it.

> Is there something I’m missing?
> 
> Also, if I do find a way to use gst_dmabuf_memory_get_fd(), how do I 
> make sure to keep a reference to the memory so that it doesn't get 
> reused after returning from the 'new-sample' callback? Is simply 
> mapping the buffer sufficient?

You won't map it.

While you hold a reference of the GstBuffer structure, hopefully, it won't be reused.

You extract the GstMemory from the GstBuffer and then get the fd from it.

You can learn how it is done in glupload:

https://cgit.freedesktop.org/gstreamer/gst-plugins-base/tree/gst-libs/gst/gl/gstglupload.c#n586

> 
> Thanks,
> 
> Nico
> 
> 
> 
> -----Original Message-----
> From: gstreamer-devel <gstreamer-devel-bounces at lists.freedesktop.org> 
> On Behalf Of Víctor Jáquez
> Sent: Thursday, August 09, 2018 8:07 AM
> To: gstreamer-devel at lists.freedesktop.org
> Subject: Re: How to consume VAAPI VASurface in appsink
> 
> On Wed, 08 Aug 2018 at 23:41, Galoppo, Nico wrote:
> > I have constructed a pipeline for decoding compressed video, with an 
> > appsink element at the end. The decoding pipeline is using GPU 
> > hardware decoding through libva/gst-vaapi. I am pulling samples in the appsink "new-sample"
> > signal callback, and have confirmed that the caps include 
> > "video/x-raw(VASurface)".
> 
> -> "video/x-raw(memory:VASurface)"
> 
> > 
> > My intent is to display (and potentially process) the framebuffer 
> > contained by the VASurface. Can I somehow get access to the 
> > VASurface through the GstBuffer contained in the GstSample? If so, 
> > then I could get to a DRM prime handle for GL/CL interop.
> 
> The VASurface is in a buffer's meta. But the meta's structure is not available. You might look at the element's source and copy the structure and extract it.
> 
> Nonetheless, if you want the DRM prime handle, I guess there's a 
> direct way to do it, after the vaapipostproc set the caps feature 
> "video/x-raw(memory:DMABuf)", then you get the dmabuf fd with
> gst_dmabuf_memory_get_fd()
> 
> https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-
> base-libs/html/gst-plugins-base-libs-dmabuf.html
> 
> vmjl
> 
> > 
> > Thanks,
> > 
> > Nico
> > 
> 
> > _______________________________________________
> > gstreamer-devel mailing list
> > gstreamer-devel at lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> 
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
gstreamer-devel at lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel