[Mesa-dev] GLSL IR & TGSI on-disk shader cache

[Nicolai Hähnle]

On 10.02.2017 12:46, Timothy Arceri wrote:
>
>
> On 10/02/17 21:35, Nicolai Hähnle wrote:
>> On 08.02.2017 23:31, Eric Anholt wrote:
>>> Bas Nieuwenhuizen <bas at basnieuwenhuizen.nl> writes:
>>>
>>>> On Wed, Feb 8, 2017, at 11:25, Matt Turner wrote:
>>>>> On Wed, Feb 8, 2017 at 12:31 AM, Timothy Arceri
>>>>> <t_arceri at yahoo.com.au>
>>>>> wrote:
>>>>>> On Tue, 2017-02-07 at 23:58 +0100, Matt Turner wrote:
>>>>>>> On Tue, Feb 7, 2017 at 4:42 AM, Timothy Arceri
>>>>>>> <tarceri at itsqueeze.com
>>>>>>>> wrote:
>>>>>>>> The reason we don't just use the build time like radv is that we
>>>>>>>> will want something consistent accross distros to enable
>>>>>>>> distribution of precompiled shaders.
>>>>>>>
>>>>>>> I think I have a solution for this. I am traveling until next week,
>>>>>>> but I will send it as soon as I can.
>>>>>>
>>>>>> Are you able to provide a hint as to what you are suggesting? I've
>>>>>> just
>>>>>> pinged the llvm about creating the api to query the version. Would be
>>>>>> interested to know if your idea will make the unnecessary.
>>>>>
>>>>> Yes, (1) build with GNU ld's --build-id=sha1 flag, which embeds the
>>>>> sha1 of the object into a special .note.gnu.build-id ELF note section;
>>>>> (2) read that section using dl_iterate_phdr() and a little bit of
>>>>> parsing; (3) use sha1 plus perhaps other information to uniquely
>>>>> identify the runtime configuration.
>>>>>
>>>>> I've written an example of how to do it here:
>>>>> http://git.mattst88.com/build-id/
>>>>>
>>>>> That should definitely be better than the alternatives I've heard
>>>>> proposed (mtime of the .so, manually hashing the source directory,
>>>>> ...). I'm happy to implement this next week, or you can feel free to
>>>>> give it a shot.
>>>>
>>>> Yeah,also  just figured that out during the week. The fundamental issue
>>>> here is that the SHA1 might come out different due to compiler
>>>> differences between distros, even if they are compiling exactly the
>>>> same
>>>> version. This increases the number of versions of the cache that
>>>> have to
>>>> be distributed if the application would like to distribute precompiled
>>>> caches.
>>>
>>> I think that trying to apply precompiled binaries from one build of
>>> Mesa+LLVM to different different builds of Mesa+LLVM is going to cause
>>> us endless pain and Mesa shouldn't allow that.
>>
>> I agree.
>>
>> The people who want to distribute precompiled binaries will have to
>> set up infrastructure where they do the precompilation across all the
>> distro/build combinations that they want to support.
>
> I believe the plan is to also have them collected directly from users.

Oh $deity, please no. That's a security nightmare waiting to happen.


>> It's not even clear to me how they intend to load those precompiled
>> binaries into the system. Is e.g. Steam just going to write into
>> Mesa's cache directory? We can't really stop them if they really
>> wanted to do that, but that seems like a pain.
>
> I believe that is the plan yes, same goes for the binary drivers.
>
>>
>> I agree with Matt that this is precisely what
>> GL_ARB_get_program_binary was designed for.
>>
>> If people are worried about the download sizes caused by the many
>> build combinations, they should look into either
>>
>> (a) compression for that purpose, or
> I will likely look into this once we land the initial cache. Currently
> there is no real information on how large this repository will grow to,
> so its more of a thought that a concern at this point.
>
> I don't know all the details of the planned system but hopefully this
> helps give a bit of a picture into how it could work.

Cool, thanks.

Nicolai