Issues with VAAPI and OpenGL
Adrianne Pilleboue
adrianne.pilleboue at wintual.fr
Tue Feb 15 14:34:16 UTC 2022
Hello,
In my C++ application, I need to transfer a decoded VASurface in a
GLTexture, using glupload and a custom appsink, for 3D rendering
purpose.
Ideally, I would like a solution with zero-copy texture transfer from
VASurface to OpenGL, directly in NV12 format.
However, I've difficulties using VAAPI+OpenGL+GStreamer. For example,
the following pipeline doesn't work :
gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
qtdemux ! vaapidecodebin ! glupload ! glcolorconvert ! glimagesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got context from element 'sink': gst.gl.GLDisplay=context,
gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
Got context from element 'vaapipostproc0': gst.vaapi.Display=context,
gst.vaapi.Display=(GstVaapiDisplay)"\(GstVaapiDisplayGLX\)\
vaapidisplayglx0";
Redistribute latency...
ERROR: from element
/GstPipeline:pipeline0/GstURISourceBin:urisourcebin0/GstTypeFindElement:typefindelement0:
Internal data stream error.
Additional debug info:
gsttypefindelement.c(1228): gst_type_find_element_loop ():
/GstPipeline:pipeline0/GstURISourceBin:urisourcebin0/GstTypeFindElement:typefindelement0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to NULL ...
Freeing pipeline ...
This pipeline is also giving the same result :
gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
qtdemux ! vaapidecodebin !
"video/x-raw(meta:GstVideoGLTextureUploadMeta),format=RGBA" ! glupload !
glcolorconvert ! glimagesink
Meanwhile, the following pipelines are working, but are usable for my
project :
gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
qtdemux ! vaapidecodebin ! "video/x-raw,format=NV12" ! glupload !
glcolorconvert ! glimagesink (non zero-copy)
gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
qtdemux ! vaapidecodebin ! vaapisink (not using OpenGL)
Do you have any idea why the first pipeline is not working ?
I'm using Ubuntu 20.04, and the following version of GStreamer :
gst-launch-1.0 version 1.16.2
GStreamer 1.16.2
https://launchpad.net/distros/ubuntu/+source/gstreamer1.0
VAAPI is working fine :
vainfo
libva info: VA-API version 1.12.0
libva info: User environment variable requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_12
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.12 (libva 2.6.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics -
21.2.3 (008fb3b)
vainfo: Supported profile and entrypoints
VAProfileNone : VAEntrypointVideoProc
VAProfileNone : VAEntrypointStats
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Simple : VAEntrypointEncSlice
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointFEI
VAProfileH264Main : VAEntrypointEncSliceLP
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointFEI
VAProfileH264High : VAEntrypointEncSliceLP
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointEncPicture
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264ConstrainedBaseline: VAEntrypointFEI
VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
VAProfileVP8Version0_3 : VAEntrypointVLD
VAProfileVP8Version0_3 : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointFEI
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD
Here is the result of gst-inspect on vaapidecodebin, for the src part.
I'm wondering why there is not DMABuf capabilities and why
GstVideoGLTextureUploadMeta is limited to RGBA format
Capabilities:
video/x-raw(memory:VASurface)
format: { (string)ENCODED, (string)NV12, (string)I420,
(string)YV12, (string)YUY2, (string)UYVY, (string)Y210,
(string)P010_10LE, (string)AYUV, (string)Y410 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
interlace-mode: progressive
video/x-raw(meta:GstVideoGLTextureUploadMeta)
format: { (string)RGBA, (string)BGRA }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
interlace-mode: progressive
video/x-raw
format: { (string)I420, (string)YV12, (string)YUY2,
(string)UYVY, (string)AYUV, (string)VUYA, (string)RGBx, (string)BGRx,
(string)xRGB, (string)xBGR, (string)RGBA, (string)BGRA, (string)ARGB,
(string)ABGR, (string)RGB, (string)BGR, (string)Y41B, (string)Y42B,
(string)YVYU, (string)Y444, (string)v210, (string)v216, (string)Y210,
(string)Y410, (string)NV12, (string)NV21, (string)GRAY8,
(string)GRAY16_BE, (string)GRAY16_LE, (string)v308, (string)RGB16,
(string)BGR16, (string)RGB15, (string)BGR15, (string)UYVP, (string)A420,
(string)RGB8P, (string)YUV9, (string)YVU9, (string)IYU1, (string)ARGB64,
(string)AYUV64, (string)r210, (string)I420_10BE, (string)I420_10LE,
(string)I422_10BE, (string)I422_10LE, (string)Y444_10BE,
(string)Y444_10LE, (string)GBR, (string)GBR_10BE, (string)GBR_10LE,
(string)NV16, (string)NV24, (string)NV12_64Z32, (string)A420_10BE,
(string)A420_10LE, (string)A422_10BE, (string)A422_10LE,
(string)A444_10BE, (string)A444_10LE, (string)NV61, (string)P010_10BE,
(string)P010_10LE, (string)IYU2, (string)VYUY, (string)GBRA,
(string)GBRA_10BE, (string)GBRA_10LE, (string)BGR10A2_LE,
(string)GBR_12BE, (string)GBR_12LE, (string)GBRA_12BE,
(string)GBRA_12LE, (string)I420_12BE, (string)I420_12LE,
(string)I422_12BE, (string)I422_12LE, (string)Y444_12BE,
(string)Y444_12LE, (string)GRAY10_LE32, (string)NV12_10LE32,
(string)NV16_10LE32, (string)NV12_10LE40 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
interlace-mode: progressive
And the same for the sink of glupload :
Capabilities:
video/x-raw(memory:GLMemory, meta:GstVideoOverlayComposition)
format: { (string)RGBA, (string)BGRA, (string)RGBx,
(string)BGRx, (string)ARGB, (string)ABGR, (string)xRGB, (string)xBGR,
(string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)AYUV,
(string)VUYA, (string)I420, (string)YV12, (string)NV12, (string)NV21,
(string)YUY2, (string)UYVY, (string)Y41B, (string)Y42B, (string)Y444,
(string)GRAY8, (string)GRAY16_LE, (string)GRAY16_BE, (string)ARGB64 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw(memory:DMABuf, meta:GstVideoOverlayComposition)
format: { (string)RGBA, (string)BGRA, (string)RGBx,
(string)BGRx, (string)ARGB, (string)ABGR, (string)xRGB, (string)xBGR,
(string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)AYUV,
(string)VUYA, (string)I420, (string)YV12, (string)NV12, (string)NV21,
(string)YUY2, (string)UYVY, (string)Y41B, (string)Y42B, (string)Y444,
(string)GRAY8, (string)GRAY16_LE, (string)GRAY16_BE, (string)ARGB64 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw(memory:SystemMemory, meta:GstVideoOverlayComposition)
format: { (string)RGBA, (string)BGRA, (string)RGBx,
(string)BGRx, (string)ARGB, (string)ABGR, (string)xRGB, (string)xBGR,
(string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)AYUV,
(string)VUYA, (string)I420, (string)YV12, (string)NV12, (string)NV21,
(string)YUY2, (string)UYVY, (string)Y41B, (string)Y42B, (string)Y444,
(string)GRAY8, (string)GRAY16_LE, (string)GRAY16_BE, (string)ARGB64 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw(meta:GstVideoGLTextureUploadMeta,
meta:GstVideoOverlayComposition)
format: RGBA
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw(memory:GLMemory)
format: { (string)RGBA, (string)BGRA, (string)RGBx,
(string)BGRx, (string)ARGB, (string)ABGR, (string)xRGB, (string)xBGR,
(string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)AYUV,
(string)VUYA, (string)I420, (string)YV12, (string)NV12, (string)NV21,
(string)YUY2, (string)UYVY, (string)Y41B, (string)Y42B, (string)Y444,
(string)GRAY8, (string)GRAY16_LE, (string)GRAY16_BE, (string)ARGB64 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw(memory:DMABuf)
format: { (string)RGBA, (string)BGRA, (string)RGBx,
(string)BGRx, (string)ARGB, (string)ABGR, (string)xRGB, (string)xBGR,
(string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)AYUV,
(string)VUYA, (string)I420, (string)YV12, (string)NV12, (string)NV21,
(string)YUY2, (string)UYVY, (string)Y41B, (string)Y42B, (string)Y444,
(string)GRAY8, (string)GRAY16_LE, (string)GRAY16_BE, (string)ARGB64 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw
format: { (string)RGBA, (string)BGRA, (string)RGBx,
(string)BGRx, (string)ARGB, (string)ABGR, (string)xRGB, (string)xBGR,
(string)RGB, (string)BGR, (string)RGB16, (string)BGR16, (string)AYUV,
(string)VUYA, (string)I420, (string)YV12, (string)NV12, (string)NV21,
(string)YUY2, (string)UYVY, (string)Y41B, (string)Y42B, (string)Y444,
(string)GRAY8, (string)GRAY16_LE, (string)GRAY16_BE, (string)ARGB64 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw(meta:GstVideoGLTextureUploadMeta)
format: RGBA
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
Here is also some debug stuff with GST_DEBUG=4 :
https://pastebin.com/ajza3Si5
Cheers
More information about the gstreamer-devel
mailing list