[Mesa-dev] [RFC] ralloc: use jemalloc for faster GLSL compilation

[Marek Olšák]

On Thu, Sep 29, 2016 at 4:56 PM, Emil Velikov <emil.l.velikov at gmail.com> wrote:
> On 29 September 2016 at 11:48, Marek Olšák <maraeo at gmail.com> wrote:
>> On Thu, Sep 29, 2016 at 11:20 AM, Nicolai Hähnle <nhaehnle at gmail.com> wrote:
>>> On 28.09.2016 18:49, Marek Olšák wrote:
>>>>
>>>> From: Marek Olšák <marek.olsak at amd.com>
>>>>
>>>> More info about jemalloc:
>>>>    https://github.com/jemalloc/jemalloc/wiki/History
>>>>
>>>> Average from 3 takes compiling Alien Isolation shaders from GLSL to GCN
>>>> bytecode:
>>>>    glibc:    17.183s
>>>>    jemalloc: 15.558s
>>>>    diff:     -9.5%
>>>>
>>>> The diff is -10.5% for a full shader-db run.
>>>> ---
>>>>
>>>> TODO: The jemalloc dependency should be added to configure.ac before this.
>>>>
>>>> We can probably redirect all malloc/calloc/realloc/free calls in Mesa to
>>>> jemalloc. We can either use _mesa_jemalloc, etc. everywhere or we can
>>>> redirect calls to jemalloc using #define malloc _mesa_jemalloc, etc.
>>>>
>>>> Right now, I just use: export LDFLAGS=-ljemalloc
>>>
>>>
>>> Sounds good to me. It should probably be a configurable option, defaulting
>>> to jemalloc and failing if not available unless explicitly disabled.
>>
>> If it was a configurable option, almost nobody would use it. Let's
>> make it mandatory.
>>
> This combined with ...
>
>>>
>>> On the Gallium side of things, switching to jemalloc could be pretty
>>> straightforward via the macros in u_memory.h, once we know that they're
>>> actually used consistently (which we currently don't -- it would be nice to
>>> know how jemalloc and glibc malloc react when the calls are mixed).
>>
>> Redefining malloc/calloc/realloc/free/posix_memalign for all Mesa code
>> would be more robust.
>>
> ... this doesn't is not a wise move.
>
> Don't force jemalloc onto everyone without having an explicit ACK from
> a wide audience, please ? Considering the static/shared link (or w/o
> jemalloc all together) distributions will have their
> preferences/policies which won't align with my/your view.

I guess we can have an option to disable jemalloc, but only if most
users won't use that option. The real problem is that the GLSL
compiler is alloc-bound and anybody wanting to use the GLSL compiler
should stay away from glibc's allocator.

The GLSL compiler can be slowed down significantly by keeping 5x
LLVMContext in memory between compilations in radeonsi. The fact that
radeonsi can indirectly slow down the GLSL compiler (but not LLVM) is
a strong indication that we have a problem.

Since I happen to be responsible for delivering a well-performing
driver for our hw across all installations, I guess you understand why
I'm inclined to making jemalloc mandatory for ralloc at least. Using
jemalloc for whole Mesa is not important and that idea can be dropped.

One comment on distributions. Some distributions complained years ago
that I made LLVM mandatory for r300g even though it worked fine
without it (but too slow on some GPUs). I made it clear that if they
didn't like the LLVM dependency, they were free to stop shipping that
driver. That taught me that distributions are not always interested in
delivering Mesa in the configuration and level of quality that was
intended by Mesa developers. If there is an option to make something
slower or worse, distributions shouldn't have the option, ever.

Marek