[Mesa-dev] [RFC 00/20] shader compiler cache

[Tapani Pälli]

Hello;

This series provides a disk cache for the shader compiler and is used 
to 'skip glLinkProgram' like GL_ARB_get_program_binary does but under 
the hood without api for the client.

Many of the patches are from my 'GL_ARB_get_program_binary' series and 
some were reviewed by Paul earlier. I've tried to split the whole thing 
in smaller pieces when possible and added more validation in IR level 
and also higher level. Most importantly sizeof(long) and some important 
structure sizes must match, otherwise cache gets invalidated.

There are some new 'uncomfortable' things that Mesa must do when having 
a disk cache. This includes directory creation, cache size management 
and just general file read/write. Some of these might not have been 
written in a portable way (especially patch 20), feedback appreciated.

This set passes all of other Piglit quick tests but not the transform 
feedback tests. I believe most (if not all) of them can be fixed by 
having key generation based on some of the variables used for xfb in 
gl_shader_program and also serializing xfb related structures but I 
have not done this yet as I think the key generation overall would need 
some better algorithm based on the gl_shader_program contents (only 
those contents which are always same for the same shader program), 
would be good to get some feedback for this. Now these contents seem a 
bit scattered, hard to generate a key.

I've limiting the supported set of shaders by having additional checks 
during shader serialization. This is simply because of some things may 
not be implemented yet or not having good test cases, these will be 
improved.

With the series I get 50% improvement for L4D2 startup (goes down from 
30s to ~15s) This is the time after startup video to menu screen. With 
some other apps the improvement varies a lot (as example I measured 
~10% improvement with GLBenchmark 2.7). It would be nice to get some 
more test results, especially with modern games.

The GL_ARB_get_program_binary extension can be implemented easily on 
top of this. Also shader backend assembly and other additional data can 
be added later as part of the same cache.

If no objections I would be tempted to start pushing in patches that 
were already reviewed as this is a lot of code to carry around.

Any comments welcome;


Tapani Pälli (20):
  glsl: memory_writer helper class for data serialization
  glsl: glsl_type serialization
  glsl: serialize methods for IR instructions
  glsl: memory_map helper class for data deserialization
  glsl: add MESA_SHADER_CACHE_MAGIC string for shader binary cache
  glsl: export populate_symbol_table function
  glsl: add MAX_NUM_STATE_SLOTS and check against builtin uniforms
  glsl: glsl_type deserialization
  glsl: ir_deserializer class for the binary shader cache
  mesa: iterate method for string_to_uint_map
  glsl: functions to serialize gl_shader and gl_shader_program
  mesa: add _Linked helper to gl_shader_program struct
  glsl: functions to deserialize gl_shader and gl_shader_program
  mesa: add _mesa_mkdir helper to imports.h
  glsl: add capability to read/map files to memory_map
  glsl: add method to deserialize binary program from given path
  mesa: add helpers for the binary shader cache
  mesa/program: add disk cache functionality
  mesa: take shader program cache in to use
  mesa: binary cache size management

 src/glsl/Makefile.sources         |   4 +
 src/glsl/builtin_variables.cpp    |   3 +
 src/glsl/glsl_types.cpp           | 164 +++++++++
 src/glsl/glsl_types.h             |  25 ++
 src/glsl/ir.h                     |  45 +++
 src/glsl/ir_deserializer.cpp      | 730 ++++++++++++++++++++++++++++++++++++++
 src/glsl/ir_deserializer.h        | 121 +++++++
 src/glsl/ir_serialize.cpp         | 340 ++++++++++++++++++
 src/glsl/ir_serialize.h           |  36 ++
 src/glsl/linker.cpp               |   2 +-
 src/glsl/linker.h                 |   3 +
 src/glsl/memory_map.h             | 230 ++++++++++++
 src/glsl/memory_writer.h          | 204 +++++++++++
 src/glsl/shader_cache.h           | 101 ++++++
 src/glsl/shader_cache_magic.h     |  36 ++
 src/glsl/shader_deserialize.cpp   | 421 ++++++++++++++++++++++
 src/glsl/shader_serialize.cpp     | 240 +++++++++++++
 src/mesa/Makefile.sources         |   1 +
 src/mesa/main/config.h            |   4 +
 src/mesa/main/context.c           |   4 +
 src/mesa/main/imports.h           |  18 +
 src/mesa/main/mtypes.h            |   7 +
 src/mesa/program/hash_table.h     |   8 +
 src/mesa/program/ir_to_mesa.cpp   |   9 +
 src/mesa/program/prog_diskcache.c | 320 +++++++++++++++++
 src/mesa/program/prog_diskcache.h |  40 +++
 26 files changed, 3115 insertions(+), 1 deletion(-)
 create mode 100644 src/glsl/ir_deserializer.cpp
 create mode 100644 src/glsl/ir_deserializer.h
 create mode 100644 src/glsl/ir_serialize.cpp
 create mode 100644 src/glsl/ir_serialize.h
 create mode 100644 src/glsl/memory_map.h
 create mode 100644 src/glsl/memory_writer.h
 create mode 100644 src/glsl/shader_cache.h
 create mode 100644 src/glsl/shader_cache_magic.h
 create mode 100644 src/glsl/shader_deserialize.cpp
 create mode 100644 src/glsl/shader_serialize.cpp
 create mode 100644 src/mesa/program/prog_diskcache.c
 create mode 100644 src/mesa/program/prog_diskcache.h

-- 
1.8.3.1