[Mesa-dev] shader cache backward compatibility

[Timothy Arceri]

On 31/08/18 19:40, Bas Nieuwenhuizen wrote:
> +TImothy
> 
> On Fri, Aug 31, 2018 at 11:32 AM Alexander Larsson <alexl at redhat.com> wrote:
>>
>> Hi, I'm the developer behind flatpak (https://flatpak.org/) and we've
>> recently run into some issues with the mesa shader cache. Flatpak has
>> a app/runtime split where the runtime is shared between app and
>> provides /usr. The runtime contains a version of mesa, but this can be
>> overridden by runtime extensions to add other OpenGL drivers.
>>
>> Each app has a separate $XDG_CACHE_HOME, pointing into the per-app
>> writable storage. For example, gedit has
>> XDG_CACHE_HOME="/home/alex/.var/app/org.gnome.gedit/cache". This
>> causes mesa to store the shader cache per-app in
>> $XDG_CACHE_HOME/mesa_shader_cache.
>>
>> In the regular case this works fine, but sometimes the version of mesa
>> is changed, with the shader cache being left in place. For example,
>> sometimes we update the mesa version in the runtime, and sometimes the
>> app switches to a new runtime which has a different mesa version.
>>
>> Such updates have caused a lot of issues for us, ranging from direct
>> crashes at startup as in
>> https://github.com/flatpak/flatpak/issues/2052 and sometimes just
>> super-slow performance. In all cases, blowing away the shader cache
>> directory fixed all issues.
>>
>> The steam flatpak has a bunch of workaround for the cache:
>>    https://github.com/flathub/com.valvesoftware.Steam/blob/master/steam_wrapper/steam_wrapper.py#L35
>> But we can't expect every app to do this.
>>
>> So, my question is, is the cache supposed to be backward compatible,
>> or at least versioned? Are we missing something in our mesa builds to
>> make that work? Is this fixed somewhere with a patch i can backport?
>> And if not, do we need to add some magic to flatpak to automatically
>> clean up the shader cache after an update?
> 
> It is supposed to be versioned automatically by mesa.
> 

Hi Alexander,

We depend on build timestamps of the mesa/llvm binaries when generating 
the sha for cache items. So if flatpak results in two versions of mesa 
having the same timestamp then there is likely going to be issues.

static inline bool
disk_cache_get_function_timestamp(void *ptr, uint32_t* timestamp)
{
    Dl_info info;
    struct stat st;
    if (!dladdr(ptr, &info) || !info.dli_fname) {
       return false;
    }
    if (stat(info.dli_fname, &st)) {
       return false;
    }
    *timestamp = st.st_mtime;
    return true;
}

Tim

>>
>> --
>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>>   Alexander Larsson                                Red Hat, Inc
>>         alexl at redhat.com         alexander.larsson at gmail.com
>> _______________________________________________
>> mesa-dev mailing list
>> mesa-dev at lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/mesa-dev