[Libva] Huge memory leak in SandyBridge while decoding h264 (big buck bunny)

[Gwenole Beauchesne]

Hi,

2012/7/2 Josep Torra <n770galaxy at gmail.com>:

> For long I'm aware of certain ambiguity in libva regarding how
> vaBuffers have to be handled.

I have to concur.

> Which make sense because there's no way for the vaapi client to know
> when a buffer is no longer in use when it's given for processing to
> vaRenderPicture. To clarify I understand the behaviour in the API as
> vaRenderPicture takes the ownership of the buffer in terms of
> refcounting and the driver is responsible to release it when it's no
> longer needed.

Actually, implementing this behaviour is not flexible enough for new
purposes like VPP or ENC. We have considered to fix the documentation
the following way: "The VA buffer is live until the vaEndPicture()
call following its use in a prior vaRenderPicture(), or until it is
explicitly destroyed through vaDestroyBuffer()".

> While the design is correct in my understanding the unfortunate thing
> is that IEGD/EMGD drivers wasn't implementing this behaviour(API)
> properly and huge memory leaks was exhibited when those drivers were
> used. Also I want to remark that the PSB driver did it properly and
> wasn't leaking last time I've checked.

Yes,
- the EMGD driver used to incorrectly implement the documented behaviour,
- the PSB driver puts the VA buffer into a dirty list of buffers that
could be re-used later, it's better but presents other issues
- the Gen (i965) driver also incorrectly implements the documented
behaviour. A correct fix was planned for 1.1.x series along with some
internal infrastructure work.

> I think that the i965 driver used to implement the correct behaviour
> prior to 1.0.15 or maybe I didn't notice the leaks when I tried.

The internal buffer is correctly refcount'ed but the VA object wrapper
is not destroyed, and never was, unless explicitly. That's why client
applications always destroyed buffers explicitly. If you check the
commit log message for the gst-vaapi change you mentioned, I have
tried to explain the status, without explicitly naming the drivers,
though. :)

> We at Fluendo added such kind of workaround conditionally to the
> IEGD/EMGD drivers as we try to respect the API as it's documented.
> Recently we received some claims of huge memory leaks while decoding
> the big buck bunny on a SNB platform (triggering the OOM killer).

If you don't use the same workaround for i965, then you are bound to
see memory leaks. That's a known problem, and I am sorry it's still
not fixed.

> From Valgrind I'm getting the following (big one) and some other VA
> related leaks:

Unless you patched valgrind to track drm bo's, the report won't be
meaningful. i.e. real leaks would be spread across false positives.

Regards,
Gwenole.