[Mesa-dev] Shader-cache status and transition

[Carl Worth]

Hi folks,

I've pushed the latest version of my shader-cache work to a branch named
"shader-cache" at:

	git://people.freedesktop.org/~cworth/mesa

I've rebased this against the latest master branch and verified that it
at least compiles and works with at least some trivial test programs.

I'm not attaching the code as patches sent via email, because things
really aren't ready for that yet, but instead will need a bit of
attention from someone else. There are a few FIXME comments in the code,
most of which should be quite trivial to address. Here are a couple of
less-trivial things that still need to be done:

1. Code needs to be added to fallback to recompile everything from
   source when there's a cache miss.

   The scenario here is that glCompileShader sees a sha1 of some GLSL
   source code that has been seen before so optimistically doesn't do
   any compilation. But then, (due to some intervening state change),
   the shader cache may miss when it actually does a lookup based on the
   composite sha1 that references the program keys, etc.

   In this case, we need code added to do the full recompile.

2. The functions brw_upload_programs() and upload_cached_program(), (in
   brw_state_upload.c and brw_shader_cache.c) need some refactoring.

   As-is, in the patch series, there are two significant problems here:

     i. The upload_cached_program function makes calls to the expensive
        functions brw_<stage>_populate_key. These functions are also
        called subsequently by the various brw_upload_<stage>_prog()
        functions. This should be refactored so that the populate_key
        functions are never called more than once for a single call to
        brw_upload_programs.

     ii. The upload_cached_program has a really cheesy 64-entry array
         named "been_there" which is an attempt to avoid excessive
         checks of the on-disk cache. This array should be
         eliminated. In its place, the code from brw_upload_programs
         down should be refactored to find compiled binaries in the
         following order:

         1st: The in-memory BO "cache", (which is poorly named
              here---it's more a stash of every program seen before,
              there's no replacement happening so it's really not acting
              like a cache).

         2nd: If not found there, look in the on-disk cache

         The current "been_there" array exists only because the code is
         structured to look at the on-disk cache first, and that will be
         really wasteful if the program happens to be in memory
         already. The right fix is to simply do the expensive checks
         only if the cheap checks fail.

         I had made several preliminary attempts at this particular
         refactoring, but wasn't able to get any to work completely. Ken
         should be up to speed on what I was doing there.

Beyond that, there's obviously a lot of testing that will be needed
before we have assurance that the shader cache is working well, (such as
getting piglit to all pass). I've only been testing with trivial
programs so far. So I anticipate that there are lots of little pieces of
state that will needed to be saved and restored that are not currently
happening. The hardest part should be tracking down each of
these. Hopefully the actual code required in each case should be nearly
trivial. Fortunately, the hard part should be very parallelizable,
(everyone grab your favorite piglit test).

I really had wanted to get this code into better shape before now. But
the sad reality is that I don't anticipate being able to spend any more
direct time on this. I will be quite happy to answer any questions that
come up from whoever takes this code on.

Thanks for everything. I've had a lot of fun playing with mesa the last
few years, and I'm sure we'll all keep bumping into each other in
various places.

-Carl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.freedesktop.org/archives/mesa-dev/attachments/20150710/c9b674a2/attachment.sig>