[Mesa-dev] RFC: Memory allocation on Mesa

Jose Fonseca jfonseca at vmware.com
Tue May 12 14:36:53 UTC 2020

From: mesa-dev <mesa-dev-bounces at lists.freedesktop.org> on behalf of Tamminen, Eero T <eero.t.tamminen at intel.com>
Sent: Monday, May 11, 2020 21:19
To: mesa-dev at lists.freedesktop.org <mesa-dev at lists.freedesktop.org>
Subject: Re: [Mesa-dev] RFC: Memory allocation on Mesa


On Mon, 2020-05-11 at 16:13 +0000, Jose Fonseca wrote:
> Some might retort: why not just play some tricks with the linker, and
> intercept all malloc/free calls, without actually having to modify
> any source code?
> Yes, that's indeed technically feasible.  And is precisely the sort
> of trick I was planing to resort to satisfy VMware needs without
> having to further modify the source code.  But for these tricks to
> work, it is absolutely imperative that one statically links C++
> library and STL.  The problem being that if one doesn't then there's
> an imbalance: the malloc/free/new/delete calls done in inline code on
> C++ headers will be intercepted, where as malloc/free/new/delete
> calls done in code from the shared object which is not inlined will
> not, causing havoc.  This is OK for us VMware (we do it already for
> many other reasons, including avoiding DLL hell.)  But I doubt it
> will be palatable for everybody else, particularly Linux distros, to
> have everything statically linked.


I've done a lot of resource usage analysis at former job[1], but I've
never had needed anything like that.  malloc etc all reside in a
separate shared library from Mesa, so calls to them always cross
dynamic library boundary and therefore all of them can be caught with
the dynamic linker features (LD_PRELOAD, LD_AUDIT...).

True, one can easily intercept all mallocs using that sort of dynamic linking tricks, when doing full application interception.  (I even have done some of the sort on https://github.com/jrfonseca/memtrail , mostly to hunt down memory leaks on LLVM.) But the goal here is to intercept the OpenGL/Vulkan driver malloc calls alone.  Not the application mallocs.  Which is difficult to segregate when doing whole application interception.

For simplicity imagine you have only these shared objects:

   application (not controlled by us)

Now imagine you're intercepting malloc doing some sort of LD_PRELOAD interception, and malloc is called.  How do you know if it's a call done by the Vulkan driver, hence should call the callback, or one done by the application, hence not call the Vulkan allocation callback.

One can look at the caller IP address, but what if the caller is in libstdc++ which is used both by Vulkan and the app, is not immediately clear which to bill the memory.  One would need to walk back the stack completely, which is complicated and not very reliable.

Imagine one guesses wrong -- the malloc interceptor believes the malloc call is done by the Vulkan driver, and calls the application callback, which then calls malloc again, and the interceptor guesses wrong again, therefore an infinite recursion loop.

Could you be confusing this with trying to catch some Mesa specific
function, where dynamic linker can catch only calls from application to
Mesa, but not calls within Mesa library itself (as they don't cross
dynamic library boundary)?

My goal from the beginning is intercepting all mallocs/frees done by Mesa OpenGL/Vulkan driver, and only those.

Note: at least earlier, new & delete typically called malloc & free (in
addition to calling ctor & dtor), in which case you don't even need to
track them separately.  You see their usage directly from the
allocation callgraph.

        - Eero

PS. Your XDot tool was a really nice tool for viewing those call-
graphs. :-)

Thanks! :)

[1] Linux has several ready-made tools for tracking resource
allocations (several Valgrind tools, ElectricFence, Duma etc), and we
added few more at Nokia, with main one being:

(Most memorable thing was early Qt/C++ application version doing
~100000(!) allocation frees while it was initializing itself, due to
redundantly creating, localizing and removing one user view.)
Yes, indeed Linux has much better tooling for this than Windows.  Note that memory debugging on Windows is just one of our needs.  The other being able to run Mesa driver on an embedded system with fixed amount of memory (a separate budget for Mesa mallocs.)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/mesa-dev/attachments/20200512/a3aa061f/attachment-0001.htm>

More information about the mesa-dev mailing list