[Mesa-dev] [PATCH] util/disk_cache: don't require 64bit atomic operations

[Timothy Arceri]

On 06/03/17 08:39, Grazvydas Ignotas wrote:
> On Sun, Mar 5, 2017 at 10:29 PM, Matt Turner <mattst88 at gmail.com> wrote:
>> On Sun, Mar 5, 2017 at 11:14 AM, Grazvydas Ignotas <notasas at gmail.com> wrote:
>>> There are still some distributions trying to support unfortunate people
>>> with old or exotic CPUs that don't have 64bit atomic operations. Missing
>>> 64bit atomic ops can be detected at compile time, so provide a 32bit
>>> fallback and add an appropriate cache size limit ceiling for that case.
>>>
>>> Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=93089
>>> Signed-off-by: Grazvydas Ignotas <notasas at gmail.com>
>>
>> Again, thanks for working on this.
>>
>> As I alluded in the other thread, I realize that the typeless
>> p_atomics API glosses over a real problem -- that atomic operations
>> are no where close to uniformly supported across platforms. libdrm for
>> instance, provides an atomic_t type, which is a struct with a single
>> int. As far as I can tell, that's really all you can portably do
>> atomic operations on.
>>
>> So I'd like to work towards getting rid of the typeless nature of
>> p_atomic*. Until that's fixed, there's not really any way to keep out
>> code that's going to break on some platform.
>>
>> With that in mind...
>>
>>> ---
>>> no commit access
>>>
>>>  src/util/disk_cache.c | 17 ++++++++++++++++-
>>>  1 file changed, 16 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/src/util/disk_cache.c b/src/util/disk_cache.c
>>> index 4a8ee21..daa5c7b 100644
>>> --- a/src/util/disk_cache.c
>>> +++ b/src/util/disk_cache.c
>>> @@ -37,6 +37,7 @@
>>>  #include <pwd.h>
>>>  #include <errno.h>
>>>  #include <dirent.h>
>>> +#include <limits.h>
>>>  #include "zlib.h"
>>>
>>>  #include "util/crc32.h"
>>> @@ -56,6 +57,11 @@
>>>  /* The number of keys that can be stored in the index. */
>>>  #define CACHE_INDEX_MAX_KEYS (1 << CACHE_INDEX_KEY_BITS)
>>>
>>> +/* limit maximum cache size on platforms that lack atomic 64bit operations */
>>> +#ifndef __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
>>> +#define CACHE_SIZE_32BIT_LIMITED
>>> +#endif
>>> +
>>>  struct disk_cache {
>>>     /* The path to the cache directory. */
>>>     char *path;
>>> @@ -65,7 +71,11 @@ struct disk_cache {
>>>     size_t index_mmap_size;
>>>
>>>     /* Pointer to total size of all objects in cache (within index_mmap) */
>>> +#ifndef CACHE_SIZE_32BIT_LIMITED
>>>     uint64_t *size;
>>> +#else
>>> +   uint32_t *size;
>>> +#endif
>>
>> I talked with Tim on IRC briefly about this. Maybe I'm just showing my
>> ignorance, but I'm surprised that we would need more than 4GB address
>> space for the shader cache.
>
> From what I understand, it's not about the address space but about the
> total size of all cache files on disk. Deus Ex was reported to use
> ~50MiB compressed, and that's just to launch the game I guess. With
> that in mind, it should be possible to break the 4GiB barrier after
> running multiple games if not now then in near future. I think Tim
> mentioned Dolphin manages to reach the current 1GiB limit already.

Yeah Dolphin was reported to hit 1GB in a couple of hours with an 
uncompressed cache. I'm really not sure what to do but I'm not convinced 
compression is doing enough yet to limit all platforms to 4GB.

I'm looking into the functionality provided by u_queue.c to be able to 
hand compression off to a thread this may enable us to use LZMA 
compression which is reported to reduce Deus Ex to ~9MB. There is also 
the possibility to strip out more data from the cache in its current 
state. For example the blob util is currently padding thing out for 
alignment sake which may of may not be all that useful [1]. We also 
store a min of 32bits so we may be able to save more space by 
introducing smaller types to blob also.

On the flip side, it seems steam (and maybe games themselves in future?) 
will override the env var for the Mesa cache dir which would store the 
cache in the games own dir, 4GB would be fine in that case.

[1] https://lists.freedesktop.org/archives/mesa-dev/2017-March/146452.html


>
> Gražvydas
> _______________________________________________
> mesa-dev mailing list
> mesa-dev at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/mesa-dev
>