[Mesa-dev] [PATCH 00/21] Reduce ir_variable memory usage

[Ian Romanick]

On 05/30/2014 05:52 PM, Kenneth Graunke wrote:
> On Friday, May 30, 2014 11:07:03 AM Eric Anholt wrote:
>> Ian Romanick <idr at freedesktop.org> writes:
>>> This series reduces the memory usage of ir_variable quite significantly.
>>>
>>> The first couple patches add a mechanism to determine the amount of
>>> memory used by any kind of IR object.  This is used to collect the data
>>> that is shown in the commit messages through the series.
> 
> I think that talloc can give you this type of information for basically free - 
> I'd rather see an implementation of the ralloc API atop talloc, and then just 
> turn the memory debugging on.  This is quite a bit of code, and incomplete...
> 
>>> Most of the rest of the patches rearrange data or store things in
>>> smaller fields.  The two interesting "subseries" are:
>>>
>>> Patches 9 through 15 and 20 through 21: Store short variable names in
>>> otherwise "dead" space in the base class.  I didn't rebase these patches
>>> to all be together because I didn't want to re-collect all the data. :)
>>> A small amount more savings could be had here, but in the test case at
>>> hand, it didn't appear worth the effort.  Adding
>>
>> I'd rather this series didn't land.
> 
> I agree with Eric, here.  A couple of the patches are pretty obvious and 
> reasonable, but a lot of them are...quite nasty.  Mashing strings between 
> struct padding is just...eesh.  Sorry. :(

Frankly, I'm shocked about that.  People use non-dynamic memory for
short strings all the time.  It's such a common practice, that I didn't
think anyone would bat an eye...

I thought the objection would be the storing the two pointers in the
same location, but nobody said boo about that.

>> Our bad memory waste for variables is mostly due to the fact that we
>> hang our builtin variables off of the symbol table, so they never get
>> freed even after dead code elimination.  These hacks (and there are some
>> *nasty* hacks in here) are just slight improvements on that bad
>> situation.
>>
>> Instead, let's fix things to not keep the symbol table past AST time (we
>> can global walk variables quickly by looking at the top level IR nodes
>> when linking), and hang our variables off of the parser state like all
>> the other IR at compile time.
> 
> Yeah, I was surprised to find that we allocate built-in variables using the 
> symbol table as the memory context, and the symbol table lives for a long 
> time.  It really only makes sense at AST->HIR time, where scopes exist.

The problem is you can't free strings that are still referenced by the
symbol table.

> After that, we just want a convenient way to find a global variable definition 
> by name.  We can just make a function to do that, and free the whole symbol 
> table, which would free all the dead variables.

We want something faster than linear search because this is used during
linking.  I have some patches for that, but it doesn't seem that anyone
is interested.

>>> Here's the punchline.  In a trimmed trace from dota2 on 32-bit,
>>> ir_variable accounts for ~5.5MB before this series.  After this series,
>>> it accounts for only ~4.5MB.
>>>
>>> Before: IR MEM: variable usage / name / total: 4955496 915817 5871313
>>> After:  IR MEM: variable usage / name / total: 4118280 644100 4762380
>>
>> I *highly* recommend using valgrind's massif tool if you're looking at
>> memory allocation.  Here's the peak allocation of my dota2 trace:
> 
> Massif is great.  There's a nice KDE-based visualizer for it as well.

I spent all day trying to get to get it (or GDB) to work with apitrace,
flipped the table, and moved on.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freedesktop.org/archives/mesa-dev/attachments/20140530/edaa02de/attachment.sig>