[cairo] Performance suite tool improvements

[Carl Worth]

I want to make sure I keep this list posted about things that are
going on. If I do a bad job of that, please prod me to post more here.

In the past week I've continued to improve the cairo performance test
suite. I've improved cairo-perf-diff so that it always uses the latest
performance test suite even when examining old versions of cairo. It
is also now clever about caching performance data based on both the
src directory of the version being tested and the perf direction of
the performance suite being used. It uses the "tree object" hashes
that git has for each of these directories for the filename used in
the cache.

This means that as we change the performance suite, old cached results
will automatically be ignored, and when we don't change the src tree
at all, (for changes to top-level files like README, or test/, etc.),
cairo-perf-diff won't waste time computing results that it knows
should not be any different.

I've also added several options to cairo-perf (such as to control the
number of iterations and to select which performance tests should be
run). It also has a new "raw" mode, (available with "cairo-perf -r"),
which will save the timings of all samples rather than just the
summary statistics. The cairo-perf-diff-files programs can read the
summary or the raw files just fine, and it will also do the right
thing if the raw mode is used multiple times and appended to the same
file.

The one thing I didn't finish is the motivation for that. I'm still
running into the problem that cairo-perf-diff sometimes reports
"small" differences, (1.4 speedup or slowdown or so, sometimes more),
even if the tree hasn't changed at all. [That is, it did before I
added the src/ tree to the hash cache---now it doesn't report anything
if the tree hasn't changed. The point is that it's still reporting
noise as significant changes sometimes.] This is annoying as it hides
real improvements, (such as the fact that the recent floating-point
speedup led to about a 1.4 speedup for all text cases).

The caching of results makes this problem worse since I have to go
fish out the "bad" results, (which requires guessing), from the cache,
delete them, and then hope that the next results will be better. This
obviously doesn't scale well.

What I want is to first switch cairo-perf-diff to use cairo-perf's raw
mode, (that much is trivial), then I want to add a new flag to
cairo-perf-diff to make it generate additional timing even if it finds
data in the cache. To be really interesting it should do this only for
tests that aren't already considered identical by
cairo-perf-diff-files, (so that program will probably need a new
--names-only option or so to advertise that information).

The idea here is that I want to be able to run:

	cairo-perf-diff HEAD

and have it report results very quickly by just running 5 iterations
or so. Then if some of the results look fishy to me, I can just run:

	cairo-perf-diff -f HEAD

(where the -f forces it to re-run tests), and this goes even faster by
first calling "cairo-perf-diff --names-only" to find which tests are
showing differences and then passing those names to "cairo-perf -r"
and appending the results to the given files.

So then I can just run that a few times until it settles out and
things should be good. This approach should also provide better
stability because samples for a given test will be spread out more,
(currently all runs for a single test happen in a burst, so can be
impacted by some long-lasting system effect). Spreading the tests out
should help average those kinds of effects out a bit.

Anyway, that's some of the tool development I've been doing lately. If
anyone wants to help me sharpen my tools a bit and finish the tiny bit
left here, that would be great.

But the more important thing is what we've been able to accomplish
with the tools, so I'll put that in my next mail.

-Carl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.freedesktop.org/archives/cairo/attachments/20061108/043918a6/attachment.pgp