Issues with VAAPI and OpenGL
Marianna Smidth Buschle
msb at qtec.com
Wed Feb 16 10:44:21 UTC 2022
> I don't understand exactly the difference between
GstVideoGLTextureUploadMeta and DMABuf. Is GstVideoGLTextureUploadMeta
generating a copy ? And why GstVideoGLTextureUploadMeta is not
compatible with NV12 ?
I cant help you with that unfortunately, I know pretty much nothing
about GL and DMA and have only used VAAPI sparsely.
> Here is the result using GST_DEBUG=*:3,GST_CAPS:5
What I generally look for with that command is more something like this:
gst-launch-1.0 v4l2src ! pnmenc ! fakesink --gst-debug=*:3,GST_CAPS:5
0:00:00.129351972 97934 0x564cb7466e30 DEBUG GST_CAPS
gstutils.c:3109:gst_pad_query_caps:<pnmenc0:sink> query returned
video/x-raw, format=(string){ RGB, GRAY8, GRAY16_BE, GRAY16_LE },
width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ],
framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:00.129480601 97934 0x564cb7466e30 DEBUG GST_CAPS
gstpad.c:2284:gst_pad_link_check_compatible_unlocked:<pnmenc0:sink> sink
caps video/x-raw, format=(string){ RGB, GRAY8, GRAY16_BE, GRAY16_LE },
width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ],
framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:00.131267810 97934 0x564cb7452f60 DEBUG GST_CAPS
gstutils.c:3109:gst_pad_query_caps:<v4l2src0:src> query returned
video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)10/1; video/x-raw,
format=(string)YUY2, width=(int)960, height=(int)540,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)15/1; video/x-raw,
format=(string)YUY2, width=(int)848, height=(int)480,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)20/1; video/x-raw,
format=(string)YUY2, width=(int)640, height=(int)480,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; video/x-raw,
format=(string)YUY2, width=(int)640, height=(int)360,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; video/x-raw,
format=(string)YUY2, width=(int)424, height=(int)240,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; video/x-raw,
format=(string)YUY2, width=(int)352, height=(int)288,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; video/x-raw,
format=(string)YUY2, width=(int)320, height=(int)240,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; video/x-raw,
format=(string)YUY2, width=(int)320, height=(int)180,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; image/jpeg,
width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1,
framerate=(fraction)30/1; image/jpeg, width=(int)960, height=(int)540,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; image/jpeg,
width=(int)848, height=(int)480, pixel-aspect-ratio=(fraction)1/1,
framerate=(fraction)30/1; image/jpeg, width=(int)640, height=(int)480,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; image/jpeg,
width=(int)640, height=(int)360, pixel-aspect-ratio=(fraction)1/1,
framerate=(fraction)30/1; image/jpeg, width=(int)424, height=(int)240,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; image/jpeg,
width=(int)352, height=(int)288, pixel-aspect-ratio=(fraction)1/1,
framerate=(fraction)30/1; image/jpeg, width=(int)320, height=(int)240,
pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1; image/jpeg,
width=(int)320, height=(int)180, pixel-aspect-ratio=(fraction)1/1,
framerate=(fraction)30/1
0:00:00.131425569 97934 0x564cb7452f60 DEBUG GST_CAPS
gstutils.c:3155:gst_pad_peer_query_caps:<v4l2src0:src> peer query
returned video/x-raw, format=(string){ RGB, GRAY8, GRAY16_BE, GRAY16_LE
}, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ],
framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:00.131476694 97934 0x564cb7452f60 WARN basesrc
gstbasesrc.c:3127:gst_base_src_loop:<v4l2src0> error: Internal data
stream error.
0:00:00.131495621 97934 0x564cb7452f60 WARN basesrc
gstbasesrc.c:3127:gst_base_src_loop:<v4l2src0> error: streaming stopped,
reason not-negotiated (-4)
So you can see that 'pnmenc' wants one of these: RGB, GRAY8, GRAY16_BE,
GRAY16_LE
But my v4l2src only provides YUY2, so it can't work without a
videoconvert in between.
Alternatively you can try simplifying you pipeline and seeing where it
fails:
gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
qtdemux ! vaapidecodebin ! glupload ! glcolorconvert ! glimagesink
gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
qtdemux ! vaapidecodebin ! glupload ! glcolorconvert ! fakesink
gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
qtdemux ! vaapidecodebin ! glupload ! fakesink
gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
qtdemux ! vaapidecodebin ! fakesink
And swapping the vaapidecodebin to the individual elements to see if
that is the problem
gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
qtdemux ! vaapidecode ! vaapipostproc ! fakesink
gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
qtdemux ! vaapidecode ! fakesink
gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
qtdemux ! vaapidecode ! vaapipostproc ! glupload ! ...
https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-vaapi-plugins/html/gstreamer-vaapi-plugins-vaapidecodebin.html
vaapidecodebin is similarGstVaapiDecode
<https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-vaapi-plugins/html/gstreamer-vaapi-plugins-vaapidecode.html#GstVaapiDecode>,
but it is composed by the vaapidecode, aGstQueue
<https://gstreamer.freedesktop.org/usr/share/gtk-doc/html/gstreamer-plugins-1.0gstreamer-plugins-queue.html#GstQueue-struct>,
and theGstVaapiPostproc
<https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-vaapi-plugins/html/gstreamer-vaapi-plugins-vaapipostproc.html#GstVaapiPostproc>,
if it is available and functional in the setup.
It offers the functionality ofGstVaapiDecode
<https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-vaapi-plugins/html/gstreamer-vaapi-plugins-vaapidecode.html#GstVaapiDecode>and
the many options ofGstVaapiPostproc
<https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-vaapi-plugins/html/gstreamer-vaapi-plugins-vaapipostproc.html#GstVaapiPostproc>.
On 16.02.2022 11.23, Adrianne Pilleboue wrote:
> Thanks a lot for your answer !
>
> Here is the result using GST_DEBUG=*:3,GST_CAPS:5
>
> https://pastebin.com/u7ghfXVN
>
> This part catch my eyes :
>
> 0:00:00.171684713 1673 0x7f37000022a0 ERROR vaapivideomemory
> gstvaapivideomemory.c:921:allocator_configure_image_info: Cannot
> create VA image
> 0:00:00.171691844 1673 0x7f37000022a0 ERROR vaapipostproc
> gstvaapipluginbase.c:608:ensure_srcpad_allocator:<vaapipostproc0>
> failed to create src pad's allocator
> 0:00:00.171695821 1673 0x7f37000022a0 WARN basetransform
> gstbasetransform.c:992:gst_base_transform_do_bufferpool:<vaapipostproc0>
> Subclass failed to decide allocation
>
> I've also started the pipeline with GST_DEBUG=5 and got a few more
> line before this error :
>
> 0:00:00.305280570 2401 0x7ff550002120 INFO vaapipostproc
> gstvaapipluginbase.c:1196:gst_vaapi_plugin_base_set_gl_context:<vaapipostproc0>
> GL context: <glcontextglx0>
> 0:00:00.305290162 2401 0x7ff550002120 DEBUG video-info
> video-info.c:403:gst_video_info_from_caps: parsing caps
> video/x-raw(meta:GstVideoGLTextureUploadMeta), format=(string)RGBA,
> width=(int)3840, height=(int)2160, framerate=(fraction)24000/1001,
> pixel-aspect-ratio=(fraction)1/1, multiview-mode=(string)mono,
> multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono,
> interlace-mode=(string)progressive
> 0:00:00.305294676 2401 0x7ff550002120 DEBUG video-info
> video-info.c:525:gst_video_info_from_caps: no colorimetry, using default
> 0:00:00.305299484 2401 0x7ff550002120 INFO vaapipostproc
> gstvaapipluginbase.c:958:gst_vaapi_plugin_base_decide_allocation:<vaapipostproc0>
> ignoring non-VAAPI pool: <glbufferpool2>
> 0:00:00.305307739 2401 0x7ff550002120 DEBUG vaapi
> gstvaapisurface.c:423:gst_vaapi_surface_new_full: size 3840x2160,
> format RGBA, flags 0x00000000
> 0:00:00.305322458 2401 0x7ff550002120 DEBUG vaapi
> gstvaapisurface.c:212:gst_vaapi_surface_create_full: surface 0x8
> 0:00:00.305326695 2401 0x7ff550002120 DEBUG vaapi
> gstvaapisurface.c:78:gst_vaapi_surface_destroy: surface 0x8
> 0:00:00.305336482 2401 0x7ff550002120 DEBUG vaapivideomemory
> gstvaapivideomemory.c:878:allocator_configure_surface_info:<vaapivideoallocator1>
> success to set the surface format RGBA for video format RGBA with
> native uploading
> 0:00:00.305341354 2401 0x7ff550002120 DEBUG vaapi
> gstvaapiimage.c:253:gst_vaapi_image_new: format RGBA, size 3840x2160
> 0:00:00.305345504 2401 0x7ff550002120 DEBUG vaapidisplay
> gstvaapidisplay.c:724:ensure_subpicture_formats: 0 subpicture formats
> 0:00:00.305349621 2401 0x7ff550002120 DEBUG vaapi
> gstvaapiimage.c:121:gst_vaapi_image_destroy: image 0xffffffff
>
> I don't understand exactly the difference between
> GstVideoGLTextureUploadMeta and DMABuf. Is GstVideoGLTextureUploadMeta
> generating a copy ? And why GstVideoGLTextureUploadMeta is not
> compatible with NV12 ?
>
> Even if I've no choice but having a copy, I would prefer to have it in
> NV12 to reduce the size of the data. NV12 to RGB conversion will then
> be done in my shader, and only on the visible part of my video.
>
> On 16.02.2022 08:50, Marianna Smidth Buschle via gstreamer-devel wrote:
>>> Hello,
>>>
>>> In my C++ application, I need to transfer a decoded VASurface in a
>>> GLTexture, using glupload and a custom appsink, for 3D rendering
>>> purpose.
>>> Ideally, I would like a solution with zero-copy texture transfer from
>>> VASurface to OpenGL, directly in NV12 format.
>>> However, I've difficulties using VAAPI+OpenGL+GStreamer. For example,
>>> the following pipeline doesn't work :
>>>
>>> gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
>>> qtdemux ! vaapidecodebin ! glupload ! glcolorconvert ! glimagesink
>>>
>>> Setting pipeline to PAUSED ...
>>> Pipeline is PREROLLING ...
>>> Got context from element 'sink': gst.gl.GLDisplay=context,
>>> gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
>>> Got context from element 'vaapipostproc0': gst.vaapi.Display=context,
>>> gst.vaapi.Display=(GstVaapiDisplay)"\(GstVaapiDisplayGLX\)\
>>> vaapidisplayglx0";
>>> Redistribute latency...
>>> ERROR: from element
>>> /GstPipeline:pipeline0/GstURISourceBin:urisourcebin0/GstTypeFindElement:typefindelement0:
>>>
>>> Internal data stream error.
>>> Additional debug info:
>>> gsttypefindelement.c(1228): gst_type_find_element_loop ():
>>> /GstPipeline:pipeline0/GstURISourceBin:urisourcebin0/GstTypeFindElement:typefindelement0:
>>>
>>> streaming stopped, reason not-negotiated (-4)
>>> ERROR: pipeline doesn't want to preroll.
>>> Setting pipeline to NULL ...
>>> Freeing pipeline ...
>>>
>>> This pipeline is also giving the same result :
>>> gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
>>> qtdemux ! vaapidecodebin !
>>> "video/x-raw(meta:GstVideoGLTextureUploadMeta),format=RGBA" !
>>> glupload !
>>> glcolorconvert ! glimagesink
>>>
>>> Meanwhile, the following pipelines are working, but are usable for my
>>> project :
>>>
>>> gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
>>> qtdemux ! vaapidecodebin ! "video/x-raw,format=NV12" ! glupload !
>>> glcolorconvert ! glimagesink (non zero-copy)
>>> gst-launch-1.0 urisourcebin uri=file:///home/myusername/myvideo.mp4 !
>>> qtdemux ! vaapidecodebin ! vaapisink (not using OpenGL)
>>>
>>> Do you have any idea why the first pipeline is not working ?
>>> I'm using Ubuntu 20.04, and the following version of GStreamer :
>>> gst-launch-1.0 version 1.16.2
>>> GStreamer 1.16.2
>>> https://launchpad.net/distros/ubuntu/+source/gstreamer1.0
>>
>> reason not-negotiated (-4)
>>
>> So you need to debug the CAPS.
>> Try running with GST_DEBUG=*:3,GST_CAPS:5
>>
>> And see if you can figure out which CAPS are incompatible.
>>
>> Then you might want to try adding some elements with can convert/parse
>> to get the right CAPS, fx 'vaapipostproc' might help.
>> But I'm not sure if you can get zero copy.
>>
>>> Here is the result of gst-inspect on vaapidecodebin, for the src part.
>>> I'm wondering why there is not DMABuf capabilities and why
>>> GstVideoGLTextureUploadMeta is limited to RGBA format
>>
>> I can see several issues in Gitlab about DMA, fx:
>>
>> https://gitlab.freedesktop.org/gstreamer/gstreamer-vaapi/-/issues/267
>>
>> https://gitlab.freedesktop.org/gstreamer/gstreamer-vaapi/-/issues/207
>>
>> https://gitlab.freedesktop.org/gstreamer/gstreamer-vaapi/-/issues/105
>
--
Best regards / Med venlig hilsen
“Marianna Smidth Buschle”
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20220216/61d71a4f/attachment-0001.htm>
More information about the gstreamer-devel
mailing list