[Mesa-dev] [RFC 07/10] mesa: add program blob cache functionality

[Tapani Pälli]

On 01/11/2018 12:43 AM, Jordan Justen wrote:
> On 2018-01-08 23:48:19, Tapani Pälli wrote:
>> Cache set and get are called in similar fashion as what is happening
>> with disk cache. Functionality requires ARB_get_program_binary and
>> EGL_ANDROID_blob_cache support.
>>
>> Signed-off-by: Tapani Pälli <tapani.palli at intel.com>
>> ---
>>   src/mesa/Makefile.sources          |   2 +
>>   src/mesa/main/program_blob_cache.c | 141 +++++++++++++++++++++++++++++++++++++
>>   src/mesa/main/program_blob_cache.h |  48 +++++++++++++
>>   src/mesa/meson.build               |   2 +
>>   src/mesa/program/ir_to_mesa.cpp    |   9 ++-
>>   5 files changed, 201 insertions(+), 1 deletion(-)
>>   create mode 100644 src/mesa/main/program_blob_cache.c
>>   create mode 100644 src/mesa/main/program_blob_cache.h
>>
>> diff --git a/src/mesa/Makefile.sources b/src/mesa/Makefile.sources
>> index 53fa486364..bbcfdb425e 100644
>> --- a/src/mesa/Makefile.sources
>> +++ b/src/mesa/Makefile.sources
>> @@ -177,6 +177,8 @@ MAIN_FILES = \
>>          main/polygon.h \
>>          main/program_binary.c \
>>          main/program_binary.h \
>> +       main/program_blob_cache.c \
>> +       main/program_blob_cache.h \
>>          main/program_resource.c \
>>          main/program_resource.h \
>>          main/querymatrix.c \
>> diff --git a/src/mesa/main/program_blob_cache.c b/src/mesa/main/program_blob_cache.c
>> new file mode 100644
>> index 0000000000..0b3ea1a549
>> --- /dev/null
>> +++ b/src/mesa/main/program_blob_cache.c
>> @@ -0,0 +1,141 @@
>> +/*
>> + * Mesa 3-D graphics library
>> + *
>> + * Copyright (C) 2018 Intel Corporation.  All Rights Reserved.
>> + *
>> + * Permission is hereby granted, free of charge, to any person obtaining a
>> + * copy of this software and associated documentation files (the "Software"),
>> + * to deal in the Software without restriction, including without limitation
>> + * the rights to use, copy, modify, merge, publish, distribute, sublicense,
>> + * and/or sell copies of the Software, and to permit persons to whom the
>> + * Software is furnished to do so, subject to the following conditions:
>> + *
>> + * The above copyright notice and this permission notice shall be included
>> + * in all copies or substantial portions of the Software.
>> + *
>> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
>> + * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
>> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
>> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
>> + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
>> + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
>> + * OTHER DEALINGS IN THE SOFTWARE.
>> + *
>> + */
>> +
>> +#include "main/errors.h"
>> +#include "main/mtypes.h"
>> +#include "main/shaderobj.h"
>> +#include "main/program_binary.h"
>> +#include "util/mesa-sha1.h"
>> +#include "compiler/glsl/program.h"
>> +
>> +#include "program_blob_cache.h"
>> +
>> +/* This is what Android EGL defines as the maxValueSize in egl_cache_t
>> + * class implementation.
>> + */
>> +#define MAX_BLOB_SIZE 64 * 1024
>> +
>> +static void
>> +generate_sha1_string(struct gl_context *ctx, struct gl_shader_program *shProg,
>> +                     char *key)
>> +{
>> +   char *buf = create_shader_program_keystr(ctx, shProg);
>> +   struct mesa_sha1 sha_ctx;
>> +   unsigned char sha1str[20];
>> +
>> +   /* Add driver sha1 to the key string. */
>> +   uint8_t driver_sha1[20];
>> +   char driver_sha1buf[41];
>> +
>> +   ctx->Driver.GetProgramBinaryDriverSHA1(ctx, driver_sha1);
>> +   _mesa_sha1_format(driver_sha1buf, driver_sha1);
>> +   ralloc_asprintf_append(&buf, "%s", driver_sha1buf);
>> +
>> +   _mesa_sha1_init(&sha_ctx);
>> +   _mesa_sha1_update(&sha_ctx, buf, strlen(buf));
>> +   _mesa_sha1_final(&sha_ctx, sha1str);
>> +   _mesa_sha1_format(key, sha1str);
>> +
>> +   ralloc_free(buf);
>> +}
>> +
>> +void
>> +_mesa_blob_cache_set(struct gl_context *ctx,
>> +                     struct gl_shader_program *shProg)
>> +{
>> +   assert(shProg->data->LinkStatus == linking_success);
>> +
>> +   /* ARB_get_program_binary support required. */
>> +   if (!ctx->blobCacheSet || !ctx->Driver.GetProgramBinaryDriverSHA1)
>> +      return;
>> +
>> +   /* Skip cache for fixed-function programs and programs that use
>> +    * transform feedback.
>> +    */
>> +   if (!shProg->Name || shProg->TransformFeedback.NumVarying > 0)
>> +      return;
>> +
>> +   GLint length;
>> +   _mesa_get_program_binary_length(ctx, shProg, &length);
> 
> I see this is built on GetProgramBinary. That might be fine, but what
> if we utilized the blob read/write as an alternative storage for the
> general disk shader cache?
> 
> It would allow us to skip compiling the GLSL.

I thought it before but decided to go with GetProgramBinary since it 
seemed more straightforward at that point. Hooking with disk_cache makes 
perfect sense though, I will try this out and see what I can come up with.

> I haven't looked much yet, but I didn't see anything that might
> explain the texture issue you noted. It is possible that it might go
> away if we instead shimmed this into the disk shader cache.

OK, thanks for taking a look! I've done some additional fixes in my 
'ANDROID_blob_cache_debug' branch (some xfb tests were still failing, 
needed to check xfb usage in getter).

// Tapani