[Mesa-dev] [PATCH 2/4] util/disk_cache: compress individual cache entries

[Timothy Arceri]

On 01/03/17 23:58, Steven Newbury wrote:
> On Wed, 2017-03-01 at 12:38 +0200, Eero Tamminen wrote:
>> Hi,
>>
>> On 01.03.2017 07:25, Timothy Arceri wrote:
>>> This reduces the cache size for Deus Ex from ~160M to ~30M for
>>> radeonsi.
>>>
>>> I'm also seeing the following improvements in minimum fps in the
>>> Shadow of Mordor benchmark:
>>>
>>> no-cache:                    ~10fps
>>> with-cache-no-compression:   ~15fps
>>> with-cache-and-compression:  ~20fps
>>>
>>> Note the with cache results are from the second run after closing
>>> and opening the game to avoid the in-memory cache.
>>>
>>> Since we only really care about decompression I went with
>>> Z_BEST_COMPRESSION as suggested on irc by Steinar H. Gunderson
>>> who has benchmarked decompression speeds.
>>
>> Did he tried liblzo instead of zlib?
>>
>> It should be faster than zlib while still having fairly OK
>> compression
>> ratio.
>>
> lz4hc would be perfect.  Decent compression ratio and very fast
> decompression.

So here are my current thoughts:

1) Do we care about fast compression/decompression?

Yes. However there is no indication that zlib is not already fast 
enough. The max size I've come across for a cache entry so far is ~200KB 
(most are smaller), at this size I'm not sure how useful a faster 
implementation is going to be at either compression or decompression.

If anyone has numbers for compression/decompression small files I would 
be interested to see them, but since there is currently no obvious 
performance bottlenecks it's not on my todo list to make such a 
comparison myself.

2) Do were care about compression ratio?

Yes. We currently have a default 1GB limit for the cache (users can 
override this). As well as games like Deus Ex having a large footprint, 
the Dolphin devs have reported reaching multiple GBs (uncompressed) in a 
few hours of running.

Initial testing also seems to point to smaller file sizes increasing 
performance due to reduced I/O. More thorough testing is probably needed 
to confirm this.

3) Is zlib the best implementation to use?

Maybe not, but we need a baseline for any comparisons. Based on my 
testing it seems to do a good job, and addresses a number of issues 
raised in 2) compared with no compression.

IMO we should go with zlib and people can provide future patches with 
justifications/stats for using a different library over zlib just like 
we do for any other performance based patch.

We may even end up with different options. vc4 for example might indeed 
be more concerned with (de)compression speed over file size who knows.