[Mesa-dev] GLSL IR & TGSI on-disk shader cache

[Nicolai Hähnle]

On 08.02.2017 23:31, Eric Anholt wrote:
> Bas Nieuwenhuizen <bas at basnieuwenhuizen.nl> writes:
>
>> On Wed, Feb 8, 2017, at 11:25, Matt Turner wrote:
>>> On Wed, Feb 8, 2017 at 12:31 AM, Timothy Arceri <t_arceri at yahoo.com.au>
>>> wrote:
>>>> On Tue, 2017-02-07 at 23:58 +0100, Matt Turner wrote:
>>>>> On Tue, Feb 7, 2017 at 4:42 AM, Timothy Arceri <tarceri at itsqueeze.com
>>>>>> wrote:
>>>>>> The reason we don't just use the build time like radv is that we
>>>>>> will want something consistent accross distros to enable
>>>>>> distribution of precompiled shaders.
>>>>>
>>>>> I think I have a solution for this. I am traveling until next week,
>>>>> but I will send it as soon as I can.
>>>>
>>>> Are you able to provide a hint as to what you are suggesting? I've just
>>>> pinged the llvm about creating the api to query the version. Would be
>>>> interested to know if your idea will make the unnecessary.
>>>
>>> Yes, (1) build with GNU ld's --build-id=sha1 flag, which embeds the
>>> sha1 of the object into a special .note.gnu.build-id ELF note section;
>>> (2) read that section using dl_iterate_phdr() and a little bit of
>>> parsing; (3) use sha1 plus perhaps other information to uniquely
>>> identify the runtime configuration.
>>>
>>> I've written an example of how to do it here:
>>> http://git.mattst88.com/build-id/
>>>
>>> That should definitely be better than the alternatives I've heard
>>> proposed (mtime of the .so, manually hashing the source directory,
>>> ...). I'm happy to implement this next week, or you can feel free to
>>> give it a shot.
>>
>> Yeah,also  just figured that out during the week. The fundamental issue
>> here is that the SHA1 might come out different due to compiler
>> differences between distros, even if they are compiling exactly the same
>> version. This increases the number of versions of the cache that have to
>> be distributed if the application would like to distribute precompiled
>> caches.
>
> I think that trying to apply precompiled binaries from one build of
> Mesa+LLVM to different different builds of Mesa+LLVM is going to cause
> us endless pain and Mesa shouldn't allow that.

I agree.

The people who want to distribute precompiled binaries will have to set 
up infrastructure where they do the precompilation across all the 
distro/build combinations that they want to support.

It's not even clear to me how they intend to load those precompiled 
binaries into the system. Is e.g. Steam just going to write into Mesa's 
cache directory? We can't really stop them if they really wanted to do 
that, but that seems like a pain.

I agree with Matt that this is precisely what GL_ARB_get_program_binary 
was designed for.

If people are worried about the download sizes caused by the many build 
combinations, they should look into either

(a) compression for that purpose, or

(b) an extension on top of GL_ARB_get_program_binary that allows build 
identifier strings to be queried.

Cheers,
Nicolai