[Mesa-dev] [RFC 00/20] shader compiler cache

[Benjamin Bellec]

Hello,

This patch set prevents me from launching Steam.
I'm on r600g (Radeon HD5850 Evergreen).

Here is the log :
$ MESA_DEBUG=1 LIBGL_DEBUG=1 STEAM_RUNTIME=0 steam
Running Steam on fedora 19 64-bit
STEAM_RUNTIME is disabled by the user
Installing breakpad exception handler for
appid(steam)/version(1401381906_client)
libGL: Can't open configuration file /home/benjamin/.drirc: Aucun
fichier ou dossier de ce type.
libGL: Can't open configuration file /home/benjamin/.drirc: Aucun
fichier ou dossier de ce type.
steam: ../../src/glsl/ralloc.c :84 : get_header:  l'assertion
« info->canary == 0x5A1106 » a échoué.
[2014-06-02 19:39:21] Startup - updater built May 29 2014 09:19:23
Uploading dump (out-of-process) [proxy '']
/tmp/dumps/crash_20140602193921_1.dmp
Finished uploading minidump (out-of-process): success = yes
response: CrashID=bp-147d1385-d725-4927-9082-1dd942140602
/home/benjamin/.local/share/Steam/steam.sh : ligne 755 :  5546
Abandon                 (core dumped)$STEAM_DEBUGGER
"$STEAMROOT/$PLATFORM/$STEAMEXE" "$@"
Installing bootstrap /home/benjamin/.local/share/Steam/bootstrap.tar.xz
Reset complete!
Restarting Steam by request...
Running Steam on fedora 19 64-bit
STEAM_RUNTIME is disabled by the user
Installing breakpad exception handler for
appid(steam)/version(1401381906_client)
libGL: Can't open configuration file /home/benjamin/.drirc: Aucun
fichier ou dossier de ce type.
libGL: Can't open configuration file /home/benjamin/.drirc: Aucun
fichier ou dossier de ce type.
steam: ../../src/glsl/ralloc.c :84 : get_header:  l'assertion
« info->canary == 0x5A1106 » a échoué.
ILocalize::AddFile() failed to load file
"public/steambootstrapper_english.txt".
[2014-06-02 19:39:25] Startup - updater built Feb 10 2014 16:03:16
Uploading dump (out-of-process) [proxy '']
/tmp/dumps/crash_20140602193925_1.dmp
/home/benjamin/.local/share/Steam/steam.sh : ligne 755 :  5645
Abandon                 (core dumped)$STEAM_DEBUGGER
"$STEAMROOT/$PLATFORM/$STEAMEXE" "$@"
Finished uploading minidump (out-of-process): success = yes
response: CrashID=bp-b11e236c-ada1-4479-84c5-fdd2b2140602

The "canary error" on Google points to a previously closed bug from
glsl-compiler :
https://bugs.freedesktop.org/show_bug.cgi?id=58039

I can bisect if you need it.

Regards.

Le 02/06/2014 14:05, Tapani Pälli a écrit :
> Hello;
>
> This series provides a disk cache for the shader compiler and is used 
> to 'skip glLinkProgram' like GL_ARB_get_program_binary does but under 
> the hood without api for the client.
>
> Many of the patches are from my 'GL_ARB_get_program_binary' series and 
> some were reviewed by Paul earlier. I've tried to split the whole thing 
> in smaller pieces when possible and added more validation in IR level 
> and also higher level. Most importantly sizeof(long) and some important 
> structure sizes must match, otherwise cache gets invalidated.
>
> There are some new 'uncomfortable' things that Mesa must do when having 
> a disk cache. This includes directory creation, cache size management 
> and just general file read/write. Some of these might not have been 
> written in a portable way (especially patch 20), feedback appreciated.
>
> This set passes all of other Piglit quick tests but not the transform 
> feedback tests. I believe most (if not all) of them can be fixed by 
> having key generation based on some of the variables used for xfb in 
> gl_shader_program and also serializing xfb related structures but I 
> have not done this yet as I think the key generation overall would need 
> some better algorithm based on the gl_shader_program contents (only 
> those contents which are always same for the same shader program), 
> would be good to get some feedback for this. Now these contents seem a 
> bit scattered, hard to generate a key.
>
> I've limiting the supported set of shaders by having additional checks 
> during shader serialization. This is simply because of some things may 
> not be implemented yet or not having good test cases, these will be 
> improved.
>
> With the series I get 50% improvement for L4D2 startup (goes down from 
> 30s to ~15s) This is the time after startup video to menu screen. With 
> some other apps the improvement varies a lot (as example I measured 
> ~10% improvement with GLBenchmark 2.7). It would be nice to get some 
> more test results, especially with modern games.
>
> The GL_ARB_get_program_binary extension can be implemented easily on 
> top of this. Also shader backend assembly and other additional data can 
> be added later as part of the same cache.
>
> If no objections I would be tempted to start pushing in patches that 
> were already reviewed as this is a lot of code to carry around.
>
> Any comments welcome;
>
>
> Tapani Pälli (20):
>   glsl: memory_writer helper class for data serialization
>   glsl: glsl_type serialization
>   glsl: serialize methods for IR instructions
>   glsl: memory_map helper class for data deserialization
>   glsl: add MESA_SHADER_CACHE_MAGIC string for shader binary cache
>   glsl: export populate_symbol_table function
>   glsl: add MAX_NUM_STATE_SLOTS and check against builtin uniforms
>   glsl: glsl_type deserialization
>   glsl: ir_deserializer class for the binary shader cache
>   mesa: iterate method for string_to_uint_map
>   glsl: functions to serialize gl_shader and gl_shader_program
>   mesa: add _Linked helper to gl_shader_program struct
>   glsl: functions to deserialize gl_shader and gl_shader_program
>   mesa: add _mesa_mkdir helper to imports.h
>   glsl: add capability to read/map files to memory_map
>   glsl: add method to deserialize binary program from given path
>   mesa: add helpers for the binary shader cache
>   mesa/program: add disk cache functionality
>   mesa: take shader program cache in to use
>   mesa: binary cache size management
>
>  src/glsl/Makefile.sources         |   4 +
>  src/glsl/builtin_variables.cpp    |   3 +
>  src/glsl/glsl_types.cpp           | 164 +++++++++
>  src/glsl/glsl_types.h             |  25 ++
>  src/glsl/ir.h                     |  45 +++
>  src/glsl/ir_deserializer.cpp      | 730 ++++++++++++++++++++++++++++++++++++++
>  src/glsl/ir_deserializer.h        | 121 +++++++
>  src/glsl/ir_serialize.cpp         | 340 ++++++++++++++++++
>  src/glsl/ir_serialize.h           |  36 ++
>  src/glsl/linker.cpp               |   2 +-
>  src/glsl/linker.h                 |   3 +
>  src/glsl/memory_map.h             | 230 ++++++++++++
>  src/glsl/memory_writer.h          | 204 +++++++++++
>  src/glsl/shader_cache.h           | 101 ++++++
>  src/glsl/shader_cache_magic.h     |  36 ++
>  src/glsl/shader_deserialize.cpp   | 421 ++++++++++++++++++++++
>  src/glsl/shader_serialize.cpp     | 240 +++++++++++++
>  src/mesa/Makefile.sources         |   1 +
>  src/mesa/main/config.h            |   4 +
>  src/mesa/main/context.c           |   4 +
>  src/mesa/main/imports.h           |  18 +
>  src/mesa/main/mtypes.h            |   7 +
>  src/mesa/program/hash_table.h     |   8 +
>  src/mesa/program/ir_to_mesa.cpp   |   9 +
>  src/mesa/program/prog_diskcache.c | 320 +++++++++++++++++
>  src/mesa/program/prog_diskcache.h |  40 +++
>  26 files changed, 3115 insertions(+), 1 deletion(-)
>  create mode 100644 src/glsl/ir_deserializer.cpp
>  create mode 100644 src/glsl/ir_deserializer.h
>  create mode 100644 src/glsl/ir_serialize.cpp
>  create mode 100644 src/glsl/ir_serialize.h
>  create mode 100644 src/glsl/memory_map.h
>  create mode 100644 src/glsl/memory_writer.h
>  create mode 100644 src/glsl/shader_cache.h
>  create mode 100644 src/glsl/shader_cache_magic.h
>  create mode 100644 src/glsl/shader_deserialize.cpp
>  create mode 100644 src/glsl/shader_serialize.cpp
>  create mode 100644 src/mesa/program/prog_diskcache.c
>  create mode 100644 src/mesa/program/prog_diskcache.h
>