[Mesa-dev] V2 On disk shader cache for i965

[Timothy Arceri]

V2: 
- rebased on master
- add support for encoding doubles
- create shaders correctly (call 
 _mesa_HashFindFreeKeyBlock(ctx->Shared->ShaderObjects, 1) to get a name) stops
 a crash when delete shader is called.
- renamed skip_cache params to is_cache_fallback, and fix related bug when
 disabling shader cache for xfb.

This series is based on the great work done by Carl, Kristian and others.

I've split up Carls original patches for easier review, and also merged
a number of fixes and clean-ups into his patches. However there is still
a small amount of code churn for example patches 14-17 that I decided to
keep separate to hopefully make review a little easier and preserve some
history but I could also squash them if people prefer.

For now I have left in some printf's as the feature is still disabled by
default and they are useful for debugging. I can hide them behind an
environment var if people would rather that, or just drop them completely.

There are no regressions after two runs of piglit with shader cache enabled on
my Broadwell machine.

This series enables on disk shader cache for vs+fs programs. For now
transform feedback, SSO, tes, geom and vs or fs only programs all skip
using the cache, these will be added as follow ups.

My goal with this series is to land something that passes piglit as soon
as possible so that others can begin testing and playing around with it.
I'm sure there are still many issue piglit is not exposing. Some issues
I'm aware of include using the correct source and attribute bindings if
they have been changed after linking and we fallback to a full recompile
after a cache miss.

There is some code duplication in brw_shader_cache.cpp that should be 
removed but I would like to make that a task for a follow-up series
as there is likely to be a bit of refactoring when adding the tes and 
geom stages anyway.

As stated above my main goal with this series is to land something that
passes piglit there is a number of optimisations that can still be done
such as skipping more validation and state recreation when falling back
to a full recompile but I would rather leave this until we have
something fully working.

Here are the shader-db times:

Cache disabled:

Thread 1 took 1360.47 seconds and compiled 13015 shaders (not including
SIMD16) with 50 GL context switches
Thread 3 took 1349.85 seconds and compiled 12848 shaders (not including
SIMD16) with 40 GL context switches
Thread 2 took 1362.94 seconds and compiled 12637 shaders (not including
SIMD16) with 36 GL context switches
Thread 0 took 1352.41 seconds and compiled 12593 shaders (not including
SIMD16) with 46 GL context switches

Cache enabled first run:

Thread 1 took 1410.30 seconds and compiled 12678 shaders (not including
SIMD16) with 34 GL context switches
Thread 2 took 1421.35 seconds and compiled 12822 shaders (not including
SIMD16) with 50 GL context switches
Thread 0 took 1410.49 seconds and compiled 12999 shaders (not including
SIMD16) with 40 GL context switches
Thread 3 took 1426.67 seconds and compiled 12594 shaders (not including
SIMD16) with 48 GL context switches

Cache enabled second run:

Thread 0 took 259.84 seconds and compiled 12817 shaders (not including
SIMD16) with 40 GL context switches
Thread 3 took 257.03 seconds and compiled 12533 shaders (not including
SIMD16) with 50 GL context switches
Thread 1 took 256.18 seconds and compiled 12828 shaders (not including
SIMD16) with 40 GL context switches
Thread 2 took 261.31 seconds and compiled 12915 shaders (not including
SIMD16) with 39 GL context switches

You can find the series in the shader-cache branch of:

https://github.com/tarceri/Mesa_arrays_of_arrays.git

MESA_GLSL_CACHE_ENABLE=1 enables the cache.