[Mesa-dev] Shader-cache status and transition

[Timothy Arceri]

On Fri, 2015-07-10 at 10:53 -0700, Carl Worth wrote:
> Hi folks,
> 
> I've pushed the latest version of my shader-cache work to a branch
> named
> "shader-cache" at:
> 
> 	git://people.freedesktop.org/~cworth/mesa

Hi all,

Just a heads up in case someone has done/intended to do any work on
this since Carl finished up. I'm going to start looking into this
sometime over the next couple of weeks.

Tim

> 
> I've rebased this against the latest master branch and verified that
> it
> at least compiles and works with at least some trivial test programs.
> 
> I'm not attaching the code as patches sent via email, because things
> really aren't ready for that yet, but instead will need a bit of
> attention from someone else. There are a few FIXME comments in the
> code,
> most of which should be quite trivial to address. Here are a couple
> of
> less-trivial things that still need to be done:
> 
> 1. Code needs to be added to fallback to recompile everything from
>    source when there's a cache miss.
> 
>    The scenario here is that glCompileShader sees a sha1 of some GLSL
>    source code that has been seen before so optimistically doesn't do
>    any compilation. But then, (due to some intervening state change),
>    the shader cache may miss when it actually does a lookup based on
> the
>    composite sha1 that references the program keys, etc.
> 
>    In this case, we need code added to do the full recompile.
> 
> 2. The functions brw_upload_programs() and upload_cached_program(),
> (in
>    brw_state_upload.c and brw_shader_cache.c) need some refactoring.
> 
>    As-is, in the patch series, there are two significant problems
> here:
> 
>      i. The upload_cached_program function makes calls to the
> expensive
>         functions brw_<stage>_populate_key. These functions are also
>         called subsequently by the various brw_upload_<stage>_prog()
>         functions. This should be refactored so that the populate_key
>         functions are never called more than once for a single call
> to
>         brw_upload_programs.
> 
>      ii. The upload_cached_program has a really cheesy 64-entry array
>          named "been_there" which is an attempt to avoid excessive
>          checks of the on-disk cache. This array should be
>          eliminated. In its place, the code from brw_upload_programs
>          down should be refactored to find compiled binaries in the
>          following order:
> 
>          1st: The in-memory BO "cache", (which is poorly named
>               here---it's more a stash of every program seen before,
>               there's no replacement happening so it's really not
> acting
>               like a cache).
> 
>          2nd: If not found there, look in the on-disk cache
> 
>          The current "been_there" array exists only because the code
> is
>          structured to look at the on-disk cache first, and that will
> be
>          really wasteful if the program happens to be in memory
>          already. The right fix is to simply do the expensive checks
>          only if the cheap checks fail.
> 
>          I had made several preliminary attempts at this particular
>          refactoring, but wasn't able to get any to work completely.
> Ken
>          should be up to speed on what I was doing there.
> 
> Beyond that, there's obviously a lot of testing that will be needed
> before we have assurance that the shader cache is working well, (such
> as
> getting piglit to all pass). I've only been testing with trivial
> programs so far. So I anticipate that there are lots of little pieces
> of
> state that will needed to be saved and restored that are not
> currently
> happening. The hardest part should be tracking down each of
> these. Hopefully the actual code required in each case should be
> nearly
> trivial. Fortunately, the hard part should be very parallelizable,
> (everyone grab your favorite piglit test).
> 
> I really had wanted to get this code into better shape before now.
> But
> the sad reality is that I don't anticipate being able to spend any
> more
> direct time on this. I will be quite happy to answer any questions
> that
> come up from whoever takes this code on.
> 
> Thanks for everything. I've had a lot of fun playing with mesa the
> last
> few years, and I'm sure we'll all keep bumping into each other in
> various places.
> 
> -Carl
> _______________________________________________
> mesa-dev mailing list
> mesa-dev at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/mesa-dev