Something wrong with gstreamer-vaapi DMABuf support

[Víctor Jáquez]

On Fri, 08 Dec 2017 at 10:45, Volker Vogelhuber wrote:
> I'm currently trying to update my gstreamer version from a patched 1.8.3 to
> 1.12.2. Unfortunately some of the changes regarding DMABuf support are still
> only in master. So I updated the gstreamer-vaapi to the git master (rev.
> af0bf7212bcf7d1064e15b7cbcb1bd901f093622).
> For my project I have a DMABuf source that is feeded with FDs from another
> process and passed to vaapih264enc. For playback I need the other way round
> and have vaapih264dec to decode buffers into a DMABuf again transfered to
> another process that will import them as a GL texture. So far this did work
> with the patched 1.8.3 (thanks to patches from Nicolas Dufresne for older
> versions).
> But with the current gstreamer-vaapi it seems like the vaapipostproc is not
> working properly. With the following pipeline I get a scrambled output
> image:
> 
> gst-launch-1.0 videotestsrc num-buffers=1 !
> video/x-raw(rgb),framerate=25/1,width=640,height=360 ! vaapipostproc !
             ^^^^
          this looks wrong
> video/x-raw,width=320,height=240 ! jpegenc ! filesink location="test.jpg"

Still, running this pipeline in my gst-uninstalled setup I got this:


(gst-launch-1.0:3370): GStreamer-WARNING **: Invalid caps feature name: rgb
Setting pipeline to PAUSED ...
0:00:00.046094208  3370       0xd39890 ERROR              gldisplay gstgldisplay_wayland.c:131:gst_gl_display_wayland_new: Failed to open Wayland display connection.
Pipeline is PREROLLING ...
Got context from element 'vaapipostproc0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
Got context from element 'vaapipostproc0': gst.vaapi.Display=context, gst.vaapi.Display=(GstVaapiDisplay)"\(GstVaapiDisplayGLX\)\ vaapidisplayglx0";
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:00.000228539
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...


And the output looks correct.

> 
> After a lot of debugging I finally found out, that it seems to have
> something to do with the allocator used for the buffers. I guess the force
> of the DMABuf allocator in gstvaapipluginbase.c ensure_srcpad_allocator is
> the root cause. After I changed the code to:
> 
> plugin->srcpad_allocator = NULL;
> if (caps && gst_vaapi_caps_feature_contains(caps,
>                  GST_VAAPI_CAPS_FEATURE_DMABUF)) {
>     if (gst_caps_is_video_raw (caps))
>       plugin->srcpad_allocator = create_dmabuf_srcpad_allocator (plugin,
>                                     vinfo, !plugin->srcpad_can_dmabuf);
>     else {
>       plugin->srcpad_allocator =
>         create_dmabuf_srcpad_allocator (plugin, vinfo, FALSE);
>         if (!plugin->srcpad_allocator)
>           goto error_create_allocator;
>     }
> }
> 
> the vaapipostproc seems to work Ok while the vaapih264dec is still able to
> provide the DMABuf I need. I guess there is something wrong as soon as the
> VAAPI is trying to map DMABuf to anything else (VASurface/SystemMemory).
> Because as long as there is no write to system memory but only encoding from
> a DMABuf and decoding into a DMABuf that is then directly used by the GPU as
> a texture there seem to be no problem. Only if I try to use the decoded
> image and e.g. write a JPG out of it, things go wrong.
> 
> I'm not sure if the problem only exists because I only upgraded
> gstreamer-vaapi to git master and not the rest of the gstreamer libraries
> (plugins, base, etc.) but maybe someone else can try to test the pipeline
> call on the command line and check if the resulting test.jpg contains proper
> image data.

Yes, it looks the case.

vmjl