[Mesa-dev] GLSL IR & TGSI on-disk shader cache

[Matt Turner]

On Wed, Feb 8, 2017 at 11:38 AM, Bas Nieuwenhuizen
<bas at basnieuwenhuizen.nl> wrote:
> On Wed, Feb 8, 2017, at 11:25, Matt Turner wrote:
>> On Wed, Feb 8, 2017 at 12:31 AM, Timothy Arceri <t_arceri at yahoo.com.au>
>> wrote:
>> > On Tue, 2017-02-07 at 23:58 +0100, Matt Turner wrote:
>> >> On Tue, Feb 7, 2017 at 4:42 AM, Timothy Arceri <tarceri at itsqueeze.com
>> >> > wrote:
>> >> > This series adds support for a GLSL IR level and TGSI (OpenGL/st)
>> >> > level caches.
>> >> >
>> >> > There are a few small bug fixes and a little bit of restructuring
>> >> > to the GLSL IR patches to make things work for Gallium drivers vs
>> >> > just i965 previously, but nothing too major.
>> >> >
>> >> > The biggest change is patch 35 which changes the mesa cache
>> >> > directory to be more structured allowing us to not fallback to
>> >> > recompiles at draw time if we realise a cache item was created
>> >> > with an old version of Mesa. Patch 36 also cleans up old cache
>> >> > files so we shouldn't get into a possition were we are constantly
>> >> > evicting cache items because its never cleaned out.
>> >> >
>> >> > Even with the new directory structure I've still left in the
>> >> > code that adds the Mesa version string to cache items because
>> >> > users can do things like override the OpenGL version which will
>> >> > change the output (in future we might want to just add this as
>> >> > part of the sha1 input).
>> >> >
>> >> > This series does not include the patch that adds cache support
>> >> > to the radeonsi backend, the main reason for this is that llvm
>> >> > currently doesn't allow the version to be queried at runtime
>> >> > (as far as I'm aware) although it seems like other are interested
>> >> > in this feature [1] so I will follow up on that.
>> >> >
>> >> > The reason we don't just use the build time like radv is that we
>> >> > will want something consistent accross distros to enable
>> >> > distribution of precompiled shaders.
>> >>
>> >> I think I have a solution for this. I am traveling until next week,
>> >> but I will send it as soon as I can.
>> >
>> > Are you able to provide a hint as to what you are suggesting? I've just
>> > pinged the llvm about creating the api to query the version. Would be
>> > interested to know if your idea will make the unnecessary.
>>
>> Yes, (1) build with GNU ld's --build-id=sha1 flag, which embeds the
>> sha1 of the object into a special .note.gnu.build-id ELF note section;
>> (2) read that section using dl_iterate_phdr() and a little bit of
>> parsing; (3) use sha1 plus perhaps other information to uniquely
>> identify the runtime configuration.
>>
>> I've written an example of how to do it here:
>> http://git.mattst88.com/build-id/
>>
>> That should definitely be better than the alternatives I've heard
>> proposed (mtime of the .so, manually hashing the source directory,
>> ...). I'm happy to implement this next week, or you can feel free to
>> give it a shot.
>
> Yeah,also  just figured that out during the week. The fundamental issue
> here is that the SHA1 might come out different due to compiler
> differences between distros, even if they are compiling exactly the same
> version. This increases the number of versions of the cache that have to
> be distributed if the application would like to distribute precompiled
> caches.

Isn't that a job for something like GL_ARB_get_program_binary, rather
than an automatic shader cache?