[Bug 797152] vaapidec: leak of dmabuf fds when transitioning from PLAYING to NULL state

[GStreamer (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=797152

--- Comment #4 from Sakari Kapanen <sakari.m.kapanen at gmail.com> ---
Created attachment 373676
  --> https://bugzilla.gnome.org/attachment.cgi?id=373676&action=edit
Output dump with log data from gstreamer-vaapi

Ok, so I got further in the investigation. I modified the repro a bit, made it
print the open dmabuf fd numbers and do just one cycle of PLAYING->NULL. I also
added some debug prints inside gstreamer-vaapi and enabled
GST_DEBUG="fdmemory:5,vaapivideomemory:5,2".

So it seems it is the original (not dup'ed) fds that are leaked.
gstreamer-vaapi in fact tries to release them in
gst_vaapi_buffer_proxy_release_handle, but the call to vaReleaseBufferHandle
there fails, returning the error code VA_STATUS_ERROR_INVALID_BUFFER (btw, this
only seems to be reported at GST_DEBUG=5 level by default which is kind of a
noisy level).

I digged a bit deeper inside the intel-vaapi-driver. I added some debug prints
to see where i965_ReleaseBufferHandle fails. It's the if condition here:
https://github.com/intel/intel-vaapi-driver/blob/master/src/i965_drv_video.c#L6671
that fails and returns the error code. So indeed the proxy->va_buf handle in
the call to vaReleaseBufferHandle seems to be invalid somehow.

If I detect the VA_STATUS_ERROR_INVALID_BUFFER in
gst_vaapi_buffer_proxy_release_handle and manually call
close(proxy->va_info.handle) if that occurs, the dmabufs are properly freed (of
course). However, this only is a band-aid to the issue - it would be good to
find where the va_buf ID goes invalid. Were you able to reproduce the issue?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.