[Mesa-dev] shader cache backward compatibility

Timothy Arceri tarceri at itsqueeze.com
Fri Aug 31 10:37:38 UTC 2018

On 31/08/18 20:10, Emil Velikov wrote:
> Hi Timothy,
> On 31 August 2018 at 10:57, Timothy Arceri <tarceri at itsqueeze.com> wrote:
>> On 31/08/18 19:40, Bas Nieuwenhuizen wrote:
>>> +TImothy
>>> On Fri, Aug 31, 2018 at 11:32 AM Alexander Larsson <alexl at redhat.com>
>>> wrote:
>>>> Hi, I'm the developer behind flatpak (https://flatpak.org/) and we've
>>>> recently run into some issues with the mesa shader cache. Flatpak has
>>>> a app/runtime split where the runtime is shared between app and
>>>> provides /usr. The runtime contains a version of mesa, but this can be
>>>> overridden by runtime extensions to add other OpenGL drivers.
>>>> Each app has a separate $XDG_CACHE_HOME, pointing into the per-app
>>>> writable storage. For example, gedit has
>>>> XDG_CACHE_HOME="/home/alex/.var/app/org.gnome.gedit/cache". This
>>>> causes mesa to store the shader cache per-app in
>>>> $XDG_CACHE_HOME/mesa_shader_cache.
>>>> In the regular case this works fine, but sometimes the version of mesa
>>>> is changed, with the shader cache being left in place. For example,
>>>> sometimes we update the mesa version in the runtime, and sometimes the
>>>> app switches to a new runtime which has a different mesa version.
>>>> Such updates have caused a lot of issues for us, ranging from direct
>>>> crashes at startup as in
>>>> https://github.com/flatpak/flatpak/issues/2052 and sometimes just
>>>> super-slow performance. In all cases, blowing away the shader cache
>>>> directory fixed all issues.
>>>> The steam flatpak has a bunch of workaround for the cache:
>>>> https://github.com/flathub/com.valvesoftware.Steam/blob/master/steam_wrapper/steam_wrapper.py#L35
>>>> But we can't expect every app to do this.
>>>> So, my question is, is the cache supposed to be backward compatible,
>>>> or at least versioned? Are we missing something in our mesa builds to
>>>> make that work? Is this fixed somewhere with a patch i can backport?
>>>> And if not, do we need to add some magic to flatpak to automatically
>>>> clean up the shader cache after an update?
>>> It is supposed to be versioned automatically by mesa.
>> Hi Alexander,
>> We depend on build timestamps of the mesa/llvm binaries when generating the
>> sha for cache items. So if flatpak results in two versions of mesa having
>> the same timestamp then there is likely going to be issues.
>> static inline bool
>> disk_cache_get_function_timestamp(void *ptr, uint32_t* timestamp)
>> {
>>     Dl_info info;
>>     struct stat st;
>>     if (!dladdr(ptr, &info) || !info.dli_fname) {
>>        return false;
>>     }
>>     if (stat(info.dli_fname, &st)) {
>>        return false;
>>     }
>>     *timestamp = st.st_mtime;
>>     return true;
>> }
> Have you tried using the build-id from src/util/build_id.c?

Hi Emil,

Honestly I've got no idea what that code does. Maybe someone who does 
could write patches to switch to it along with an explanation of why its 
better. Even just adding some comments in that file would be helpful.

I don't want to be the one responsible for it (and any new issues with 
the cache) when I'm not aware of how it works :(


> The reproducible builds' date/time override feature, is something
> people could easily misusing.
> Effectively having the same timestamp across multiple, completely
> different, builds.
> Emil

More information about the mesa-dev mailing list