[Mesa-dev] GLSL IR & TGSI on-disk shader cache

[Kai Wasserbäch]

Hey,
Pierre-Loup A. Griffais wrote on 11.02.2017 03:44:
> On 02/10/2017 04:01 AM, Nicolai Hähnle wrote:
>> On 10.02.2017 12:46, Timothy Arceri wrote:
>>> On 10/02/17 21:35, Nicolai Hähnle wrote:
>>> [...]
>>>
>>>> It's not even clear to me how they intend to load those precompiled
>>>> binaries into the system. Is e.g. Steam just going to write into
>>>> Mesa's cache directory? We can't really stop them if they really
>>>> wanted to do that, but that seems like a pain.
>>>
>>> I believe that is the plan yes, same goes for the binary drivers.
>>>
>>>>
>>>> I agree with Matt that this is precisely what
>>>> GL_ARB_get_program_binary was designed for.
>>>>
>>>> If people are worried about the download sizes caused by the many
>>>> build combinations, they should look into either
>>>>
>>>> (a) compression for that purpose, or
>>> I will likely look into this once we land the initial cache. Currently
>>> there is no real information on how large this repository will grow to,
>>> so its more of a thought that a concern at this point.
>>>
>>> I don't know all the details of the planned system but hopefully this
>>> helps give a bit of a picture into how it could work.
>>
>> Cool, thanks.
> 
> ARB_get_program_binary is not useful for collection/redistribution.
> 
> It seems like regardless of any external participant to the system, being robust
> against LLVM differences is something the cache needs to solve.
> 
> Ideally LLVM could expose a version string/number that's granular enough, and
> distros/users could be trusted to properly amend that version string if they
> make functional changes to something that can affect shader compilation, but
> maybe that's wishful thinking.

that version string would need to include the SVN revision, right? Otherwise all
users of LLVM trunk would be screwed*, right? At the moment that kind of
information is most often found in the package version, not what llvm-config
reports (e.g. my llvm-config says 5.0.0-devel, while it would need to say at
least something like 5.0.0-devel~svn294745). As an example, have a look at the
Debian and Ubuntu packages at <http://apt.llvm.org/> (maintained by the Debian
Developer for the in-distribution packages). And what would need to be reported
if I carry a local patch (because I might be testing a solution to a bug)? Right
now I've applied <https://reviews.llvm.org/D26348?download=true> on top of LLVM
and <https://bugs.freedesktop.org/attachment.cgi?id=127922> (see
<https://bugs.freedesktop.org/show_bug.cgi?id=97988>) as well as a revert of
7b32ae4df5bc19c378598d6a950a6019fa64ece6 (see
<https://bugs.freedesktop.org/show_bug.cgi?id=99542>) on top of Mesa. At least
the patches for bug 97988 affect the generated instructions/IR.

* Sort of depends on whether the goal here is to get rid of the GLSL-source
shaders and only ship the pre-compiled ones or if it's more of a "initial speed
bump" thing. Though I honestly can't see the former working unless you redefine
Linux support in Steam as "only with SteamOS".

> I'm not so sure that a build-id is a silver bullet here either, but if you can
> rely on its existence because it's written into the binary by a part of the
> build system that no distro will ever mess with, it seems like it would work. It
> would be more conservative than strictly needed, but that isn't a major issue in
> the short term. It would at least allow us to get compelling data showing
> whether moving to a less granular cache key would allow us to serve
> significantly more users.

BuildIDs could be working (at least on Debian and derived distributions as well
as Fedora/Red Hat, I know the BuildIDs are added). But of course, minor changes
which might not affect the usability of the cache, would still change that ID,
so the miss chance should be pretty high – especially for people following LLVM
trunk and/or Mesa master.
At the end of the day I really don't see how Steam "pre-filling" the cache is a
good idea. At least for me it's just going to be dead data more often than not,
that has to be cleaned out on top of all the ancient libraries in the Steam
runtime, which usually prevent Steam from launching with up-to-date drivers. I'd
rather have Mesa fill it on-demand. (And yes, I know most users would stay with
their distribution's versions, but then you'll have subtle differences in
applied patches to consider and I rather not have the binary versions for all of
those in my cache. Sure BuildID solves the loading/matching, but not that you
might never be able to use the pre-loaded objects.)

Cheers,
Kai

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: OpenPGP digital signature
URL: <https://lists.freedesktop.org/archives/mesa-dev/attachments/20170211/4e007d29/attachment-0001.sig>