[Mesa-dev] GLSL IR & TGSI on-disk shader cache

[Timothy Arceri]

On 10/02/17 21:35, Nicolai Hähnle wrote:
> On 08.02.2017 23:31, Eric Anholt wrote:
>> Bas Nieuwenhuizen <bas at basnieuwenhuizen.nl> writes:
>>
>>> On Wed, Feb 8, 2017, at 11:25, Matt Turner wrote:
>>>> On Wed, Feb 8, 2017 at 12:31 AM, Timothy Arceri 
>>>> <t_arceri at yahoo.com.au>
>>>> wrote:
>>>>> On Tue, 2017-02-07 at 23:58 +0100, Matt Turner wrote:
>>>>>> On Tue, Feb 7, 2017 at 4:42 AM, Timothy Arceri 
>>>>>> <tarceri at itsqueeze.com
>>>>>>> wrote:
>>>>>>> The reason we don't just use the build time like radv is that we
>>>>>>> will want something consistent accross distros to enable
>>>>>>> distribution of precompiled shaders.
>>>>>>
>>>>>> I think I have a solution for this. I am traveling until next week,
>>>>>> but I will send it as soon as I can.
>>>>>
>>>>> Are you able to provide a hint as to what you are suggesting? I've 
>>>>> just
>>>>> pinged the llvm about creating the api to query the version. Would be
>>>>> interested to know if your idea will make the unnecessary.
>>>>
>>>> Yes, (1) build with GNU ld's --build-id=sha1 flag, which embeds the
>>>> sha1 of the object into a special .note.gnu.build-id ELF note section;
>>>> (2) read that section using dl_iterate_phdr() and a little bit of
>>>> parsing; (3) use sha1 plus perhaps other information to uniquely
>>>> identify the runtime configuration.
>>>>
>>>> I've written an example of how to do it here:
>>>> http://git.mattst88.com/build-id/
>>>>
>>>> That should definitely be better than the alternatives I've heard
>>>> proposed (mtime of the .so, manually hashing the source directory,
>>>> ...). I'm happy to implement this next week, or you can feel free to
>>>> give it a shot.
>>>
>>> Yeah,also  just figured that out during the week. The fundamental issue
>>> here is that the SHA1 might come out different due to compiler
>>> differences between distros, even if they are compiling exactly the 
>>> same
>>> version. This increases the number of versions of the cache that 
>>> have to
>>> be distributed if the application would like to distribute precompiled
>>> caches.
>>
>> I think that trying to apply precompiled binaries from one build of
>> Mesa+LLVM to different different builds of Mesa+LLVM is going to cause
>> us endless pain and Mesa shouldn't allow that.
>
> I agree.
>
> The people who want to distribute precompiled binaries will have to 
> set up infrastructure where they do the precompilation across all the 
> distro/build combinations that they want to support.

I believe the plan is to also have them collected directly from users.

>
> It's not even clear to me how they intend to load those precompiled 
> binaries into the system. Is e.g. Steam just going to write into 
> Mesa's cache directory? We can't really stop them if they really 
> wanted to do that, but that seems like a pain.

I believe that is the plan yes, same goes for the binary drivers.

>
> I agree with Matt that this is precisely what 
> GL_ARB_get_program_binary was designed for.
>
> If people are worried about the download sizes caused by the many 
> build combinations, they should look into either
>
> (a) compression for that purpose, or
I will likely look into this once we land the initial cache. Currently 
there is no real information on how large this repository will grow to, 
so its more of a thought that a concern at this point.

I don't know all the details of the planned system but hopefully this 
helps give a bit of a picture into how it could work.

>
> (b) an extension on top of GL_ARB_get_program_binary that allows build 
> identifier strings to be queried.
>
> Cheers,
> Nicolai
> _______________________________________________
> mesa-dev mailing list
> mesa-dev at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/mesa-dev