[Mesa-dev] [PATCH 0/2] glsl: avoid quadratic behavior during preprocessing

[Thomas Helland]

Hi,

I haven't looked to closely into this series yet.
Just thought I'd point out this series I wrote back in May,
in case you didn't see it, or remembered it.

I haven't gotten around to finishing it (whats left is porting
the tests to gtest). The implementation should be
fine though; I've put it through quite some testing.
At a quick glance it seems this solution is more involved.

I'm not sure if my stringbuffer implementation has the
same advantage of low memory usage when running
through address sanitizer. It will likely help a bit, but not
to the extent your implementation does. What do you think?

Regards,
Thomas

2017-08-26 14:11 GMT+00:00 Nicolai Hähnle <nhaehnle at gmail.com>:
> Hi all,
>
> The GLSL preprocessor operates by successively appending preprocessed
> pieces to a string via ralloc_str_append and friends. Each of these
> functions calls realloc() every time it is used, which is potentially
> quadratic behavior.
>
> This doesn't seem to be a problem usually because the system allocator
> can avoid copies most of the time. But when building with the address
> sanitizer, realloc actually copies the memory every single time, and
> both time spent in memcpy() and total memory usage goes through the
> roof (according to top, arb_shader_atomic_counters-max-counters uses a
> cool 2.5GB of RAM when asan is enabled, which is how this came to my
> attention in the first place).
>
> This series adds a simple stringbuf utility which implements appends
> by internally keeping a linked list of buffers that are then concatenated
> once at the end.
>
> It doesn't seem to be worth the effort to modify other users of the
> ralloc append functions, since most of them only concatenate a small
> number of pieces.
>
> Please review!
> Thanks,
> Nicolai
> --
>  src/compiler/glsl/glcpp/glcpp-parse.y |  99 +++++--------
>  src/compiler/glsl/glcpp/glcpp.h       |   7 +-
>  src/compiler/glsl/glcpp/pp.c          |  45 +++---
>  src/util/Makefile.sources             |   2 +
>  src/util/stringbuf.c                  | 185 ++++++++++++++++++++++++
>  src/util/stringbuf.h                  |  97 +++++++++++++
>  6 files changed, 343 insertions(+), 92 deletions(-)
>
> _______________________________________________
> mesa-dev mailing list
> mesa-dev at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/mesa-dev