[Mesa-dev] [RFC 00/20] shader compiler cache
Tapani Pälli
tapani.palli at intel.com
Mon Jun 2 21:40:44 PDT 2014
Hi;
On 06/02/2014 08:44 PM, Benjamin Bellec wrote:
> Hello,
>
> This patch set prevents me from launching Steam.
> I'm on r600g (Radeon HD5850 Evergreen).
Thanks for testing the patches! What I forgot to state in the mail was
that I've only been able to test this with Intel i965 driver, things
might not be initialized correctly for gallium drivers. I can try to fix
this if you send me a gdb backtrace, most interesting part is that what
pointer is being given to ralloc here. There's many things allocated
with ralloc but some without and seems like a mismatch somewhere in the
code.
> Here is the log :
> $ MESA_DEBUG=1 LIBGL_DEBUG=1 STEAM_RUNTIME=0 steam
> Running Steam on fedora 19 64-bit
> STEAM_RUNTIME is disabled by the user
> Installing breakpad exception handler for
> appid(steam)/version(1401381906_client)
> libGL: Can't open configuration file /home/benjamin/.drirc: Aucun
> fichier ou dossier de ce type.
> libGL: Can't open configuration file /home/benjamin/.drirc: Aucun
> fichier ou dossier de ce type.
> steam: ../../src/glsl/ralloc.c :84 : get_header: l'assertion
> « info->canary == 0x5A1106 » a échoué.
> [2014-06-02 19:39:21] Startup - updater built May 29 2014 09:19:23
> Uploading dump (out-of-process) [proxy '']
> /tmp/dumps/crash_20140602193921_1.dmp
> Finished uploading minidump (out-of-process): success = yes
> response: CrashID=bp-147d1385-d725-4927-9082-1dd942140602
> /home/benjamin/.local/share/Steam/steam.sh : ligne 755 : 5546
> Abandon (core dumped)$STEAM_DEBUGGER
> "$STEAMROOT/$PLATFORM/$STEAMEXE" "$@"
> Installing bootstrap /home/benjamin/.local/share/Steam/bootstrap.tar.xz
> Reset complete!
> Restarting Steam by request...
> Running Steam on fedora 19 64-bit
> STEAM_RUNTIME is disabled by the user
> Installing breakpad exception handler for
> appid(steam)/version(1401381906_client)
> libGL: Can't open configuration file /home/benjamin/.drirc: Aucun
> fichier ou dossier de ce type.
> libGL: Can't open configuration file /home/benjamin/.drirc: Aucun
> fichier ou dossier de ce type.
> steam: ../../src/glsl/ralloc.c :84 : get_header: l'assertion
> « info->canary == 0x5A1106 » a échoué.
> ILocalize::AddFile() failed to load file
> "public/steambootstrapper_english.txt".
> [2014-06-02 19:39:25] Startup - updater built Feb 10 2014 16:03:16
> Uploading dump (out-of-process) [proxy '']
> /tmp/dumps/crash_20140602193925_1.dmp
> /home/benjamin/.local/share/Steam/steam.sh : ligne 755 : 5645
> Abandon (core dumped)$STEAM_DEBUGGER
> "$STEAMROOT/$PLATFORM/$STEAMEXE" "$@"
> Finished uploading minidump (out-of-process): success = yes
> response: CrashID=bp-b11e236c-ada1-4479-84c5-fdd2b2140602
>
> The "canary error" on Google points to a previously closed bug from
> glsl-compiler :
> https://bugs.freedesktop.org/show_bug.cgi?id=58039
>
> I can bisect if you need it.
>
> Regards.
>
> Le 02/06/2014 14:05, Tapani Pälli a écrit :
>> Hello;
>>
>> This series provides a disk cache for the shader compiler and is used
>> to 'skip glLinkProgram' like GL_ARB_get_program_binary does but under
>> the hood without api for the client.
>>
>> Many of the patches are from my 'GL_ARB_get_program_binary' series and
>> some were reviewed by Paul earlier. I've tried to split the whole thing
>> in smaller pieces when possible and added more validation in IR level
>> and also higher level. Most importantly sizeof(long) and some important
>> structure sizes must match, otherwise cache gets invalidated.
>>
>> There are some new 'uncomfortable' things that Mesa must do when having
>> a disk cache. This includes directory creation, cache size management
>> and just general file read/write. Some of these might not have been
>> written in a portable way (especially patch 20), feedback appreciated.
>>
>> This set passes all of other Piglit quick tests but not the transform
>> feedback tests. I believe most (if not all) of them can be fixed by
>> having key generation based on some of the variables used for xfb in
>> gl_shader_program and also serializing xfb related structures but I
>> have not done this yet as I think the key generation overall would need
>> some better algorithm based on the gl_shader_program contents (only
>> those contents which are always same for the same shader program),
>> would be good to get some feedback for this. Now these contents seem a
>> bit scattered, hard to generate a key.
>>
>> I've limiting the supported set of shaders by having additional checks
>> during shader serialization. This is simply because of some things may
>> not be implemented yet or not having good test cases, these will be
>> improved.
>>
>> With the series I get 50% improvement for L4D2 startup (goes down from
>> 30s to ~15s) This is the time after startup video to menu screen. With
>> some other apps the improvement varies a lot (as example I measured
>> ~10% improvement with GLBenchmark 2.7). It would be nice to get some
>> more test results, especially with modern games.
>>
>> The GL_ARB_get_program_binary extension can be implemented easily on
>> top of this. Also shader backend assembly and other additional data can
>> be added later as part of the same cache.
>>
>> If no objections I would be tempted to start pushing in patches that
>> were already reviewed as this is a lot of code to carry around.
>>
>> Any comments welcome;
>>
>>
>> Tapani Pälli (20):
>> glsl: memory_writer helper class for data serialization
>> glsl: glsl_type serialization
>> glsl: serialize methods for IR instructions
>> glsl: memory_map helper class for data deserialization
>> glsl: add MESA_SHADER_CACHE_MAGIC string for shader binary cache
>> glsl: export populate_symbol_table function
>> glsl: add MAX_NUM_STATE_SLOTS and check against builtin uniforms
>> glsl: glsl_type deserialization
>> glsl: ir_deserializer class for the binary shader cache
>> mesa: iterate method for string_to_uint_map
>> glsl: functions to serialize gl_shader and gl_shader_program
>> mesa: add _Linked helper to gl_shader_program struct
>> glsl: functions to deserialize gl_shader and gl_shader_program
>> mesa: add _mesa_mkdir helper to imports.h
>> glsl: add capability to read/map files to memory_map
>> glsl: add method to deserialize binary program from given path
>> mesa: add helpers for the binary shader cache
>> mesa/program: add disk cache functionality
>> mesa: take shader program cache in to use
>> mesa: binary cache size management
>>
>> src/glsl/Makefile.sources | 4 +
>> src/glsl/builtin_variables.cpp | 3 +
>> src/glsl/glsl_types.cpp | 164 +++++++++
>> src/glsl/glsl_types.h | 25 ++
>> src/glsl/ir.h | 45 +++
>> src/glsl/ir_deserializer.cpp | 730 ++++++++++++++++++++++++++++++++++++++
>> src/glsl/ir_deserializer.h | 121 +++++++
>> src/glsl/ir_serialize.cpp | 340 ++++++++++++++++++
>> src/glsl/ir_serialize.h | 36 ++
>> src/glsl/linker.cpp | 2 +-
>> src/glsl/linker.h | 3 +
>> src/glsl/memory_map.h | 230 ++++++++++++
>> src/glsl/memory_writer.h | 204 +++++++++++
>> src/glsl/shader_cache.h | 101 ++++++
>> src/glsl/shader_cache_magic.h | 36 ++
>> src/glsl/shader_deserialize.cpp | 421 ++++++++++++++++++++++
>> src/glsl/shader_serialize.cpp | 240 +++++++++++++
>> src/mesa/Makefile.sources | 1 +
>> src/mesa/main/config.h | 4 +
>> src/mesa/main/context.c | 4 +
>> src/mesa/main/imports.h | 18 +
>> src/mesa/main/mtypes.h | 7 +
>> src/mesa/program/hash_table.h | 8 +
>> src/mesa/program/ir_to_mesa.cpp | 9 +
>> src/mesa/program/prog_diskcache.c | 320 +++++++++++++++++
>> src/mesa/program/prog_diskcache.h | 40 +++
>> 26 files changed, 3115 insertions(+), 1 deletion(-)
>> create mode 100644 src/glsl/ir_deserializer.cpp
>> create mode 100644 src/glsl/ir_deserializer.h
>> create mode 100644 src/glsl/ir_serialize.cpp
>> create mode 100644 src/glsl/ir_serialize.h
>> create mode 100644 src/glsl/memory_map.h
>> create mode 100644 src/glsl/memory_writer.h
>> create mode 100644 src/glsl/shader_cache.h
>> create mode 100644 src/glsl/shader_cache_magic.h
>> create mode 100644 src/glsl/shader_deserialize.cpp
>> create mode 100644 src/glsl/shader_serialize.cpp
>> create mode 100644 src/mesa/program/prog_diskcache.c
>> create mode 100644 src/mesa/program/prog_diskcache.h
>>
More information about the mesa-dev
mailing list