Something wrong with gstreamer-vaapi DMABuf support

[Volker Vogelhuber]

I'm currently trying to update my gstreamer version from a patched 1.8.3 
to 1.12.2. Unfortunately some of the changes regarding DMABuf support 
are still only in master. So I updated the gstreamer-vaapi to the git 
master (rev. af0bf7212bcf7d1064e15b7cbcb1bd901f093622).
For my project I have a DMABuf source that is feeded with FDs from 
another process and passed to vaapih264enc. For playback I need the 
other way round and have vaapih264dec to decode buffers into a DMABuf 
again transfered to another process that will import them as a GL 
texture. So far this did work with the patched 1.8.3 (thanks to patches 
from Nicolas Dufresne for older versions).
But with the current gstreamer-vaapi it seems like the vaapipostproc is 
not working properly. With the following pipeline I get a scrambled 
output image:

gst-launch-1.0 videotestsrc num-buffers=1 ! 
video/x-raw(rgb),framerate=25/1,width=640,height=360 ! vaapipostproc ! 
video/x-raw,width=320,height=240 ! jpegenc ! filesink location="test.jpg"

After a lot of debugging I finally found out, that it seems to have 
something to do with the allocator used for the buffers. I guess the 
force of the DMABuf allocator in gstvaapipluginbase.c 
ensure_srcpad_allocator is the root cause. After I changed the code to:

plugin->srcpad_allocator = NULL;
if (caps && gst_vaapi_caps_feature_contains(caps,
                  GST_VAAPI_CAPS_FEATURE_DMABUF)) {
     if (gst_caps_is_video_raw (caps))
       plugin->srcpad_allocator = create_dmabuf_srcpad_allocator (plugin,
                                     vinfo, !plugin->srcpad_can_dmabuf);
     else {
       plugin->srcpad_allocator =
         create_dmabuf_srcpad_allocator (plugin, vinfo, FALSE);
         if (!plugin->srcpad_allocator)
           goto error_create_allocator;
     }
}

the vaapipostproc seems to work Ok while the vaapih264dec is still able 
to provide the DMABuf I need. I guess there is something wrong as soon 
as the VAAPI is trying to map DMABuf to anything else 
(VASurface/SystemMemory). Because as long as there is no write to system 
memory but only encoding from a DMABuf and decoding into a DMABuf that 
is then directly used by the GPU as a texture there seem to be no 
problem. Only if I try to use the decoded image and e.g. write a JPG out 
of it, things go wrong.

I'm not sure if the problem only exists because I only upgraded 
gstreamer-vaapi to git master and not the rest of the gstreamer 
libraries (plugins, base, etc.) but maybe someone else can try to test 
the pipeline call on the command line and check if the resulting 
test.jpg contains proper image data.