[Mesa-dev] [PATCH 2/3] st: Sweep NIR after linking phase to free held memory

[Danylo Piliaiev]

On 19.07.18 12:30, Timothy Arceri wrote:
>> On 19.07.18 11:47, Timothy Arceri wrote:
>>> On 19/07/18 08:31, Eric Anholt wrote:
>>>> Danylo Piliaiev <danylo.piliaiev at gmail.com> writes:
>>>>
>>>>> After optimization passes and many trasfromations most of memory
>>>>
>>>> "transformations"
>>>>
>>>>> NIR holds is a garbage which was being freed only after shader 
>>>>> deletion.
>>>>
>>>> "is garbage"
>>>>
>>>>> Freeing it at the end of linking will save memory which would be 
>>>>> useful
>>>>> in case there are a lot of complex shaders being compiled.
>>>>> The common case for this issue is 32bit game running under Wine.
>>>>>
>>>>> The cost of the optimization is around ~3-5% of compilation speed
>>>>> with complex shaders.
>>>>>
>>>>> Signed-off-by: Danylo Piliaiev <danylo.piliaiev at globallogic.com>
>>>>
>>>> This seems good, and I'm running it through the CTS now.
>>>>
>>>
>>> The problem is this does the sweep too early. We still do lots of 
>>> work on NIR after this, I've thought about this a few times and it 
>>> really seems we should be able to call a driver specific function 
>>> from the st and pass it the IR so it can do what ever it wants 
>>> (lowering opts etc) and spits it back out. Once this is done we 
>>> should then call sweep and cache the IR.
>>>
>>> At the very least we should call sweep before we cache NIR rather 
>>> than where this patch places it.
>> I debugged mesa once more and cannot see where else memory is 
>> allocated for NIR after this sweep. Could you point it to me? After 
>> this sweep the memory NIR holds never grows. Later this NIR is cloned 
>> from and these cloned NIRs are being sweeped in other place.
>
> Ah yes you are right. We do clone it when creating variants I'd 
> forgotten about that. In that case please ignore my comment.

Good, I thought I really missed something...

Also during my investigation of Mesa memory usage I wrote gdb pretty 
printer which shows how much memory variable holds in its ralloc context 
(with all its children), it was crudely written and at this moment has 
sever limitation: x64 only, depends on internal malloc implementation 
and other hardcoded things, also I wasn't able to nicely display 
children of a variable. The reason that pretty printer was done this way 
is that calling c function (e.g. malloc_usable_size) corrupts backtrace 
somehow.

Example usage:

(gdb) source ralloc_info_pretty_printer.py
(gdb) backtrace
#0  brw_link_shader (ctx=0x5555558ca0a0 <size: 96432>, 
shProg=0x555555d850c0 <size: 528>) at brw_link.cpp:320
#1  0x00007ffff2732b6f in _mesa_glsl_link_shader (ctx=0x5555558ca0a0 
<size: 96432>, prog=0x555555d850c0 <size: 528>) at 
program/ir_to_mesa.cpp:3174
#2  0x00007ffff25a1862 in link_program (no_error=false, 
shProg=0x555555d850c0 <size: 528>, ctx=0x5555558ca0a0 <size: 96432>) at 
main/shaderapi.c:1206
#3  link_program_error (ctx=0x5555558ca0a0 <size: 96432>, 
shProg=0x555555d850c0 <size: 528>) at main/shaderapi.c:1286
#4  0x00007ffff25a2f00 in _mesa_LinkProgram (programObj=3) at 
main/shaderapi.c:1778
#5  0x0000555555556de1 in main () at 
/home/danylo/Projects/shader_compilation_memory_test/test.cpp:421
(gdb) p prog->nir
$1 = 0x555555dc2b20 <size: 241296>

If there is any interest in having it in Mesa I can clean it up. You can 
find its code in attachment.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: ralloc_info_pretty_printer.py
Type: text/x-python
Size: 2662 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/mesa-dev/attachments/20180719/022e1f24/attachment-0001.py>