the mysterious cairo-perf (was: Re: [cairo] [PATCH] Performance patch for bo intersections)

[Carl Worth]

On Wed, 13 Dec 2006 22:22:44 +0200 (EET), M Joonas Pihlaja wrote:
> - cairo-perf is discarding the runs that it thinks went too
> quickly.  This seems slightly odd to me -- aren't the runs that
> went the fastest the ones that have the least error in them?
> (assuming we can trust gettimeofday())
>
> Having said that, the shorter runs definitely seem to be the ones
> that are likely the most sensitive to conspiring accidents of
> memory or code layout, and all that other nasty stuff which we
> really don't have any control over in the source code.

That's the idea.

If all the planets align and you magically get a super-fast result on
one test, we really don't want to use that value to compare one run
against another, (since it's not the alignment of the planets we're
trying to measure, but the effects of the patch).

So it actually _can_ make sense to discard even a very fast value if
we can statistically identify it as an outlier. What might be
interesting is to explore how often we actually get such an outlier,
(I imagine it's usually not discarding anything on this end---but I
have occasionally seen entirely inexplicable results that are
obviously "too fast" to be "real".)

I'm always open to any improvements to make the measurements here more
reliable.

What I've found so far is that doing a lot of iterations helps, but it
takes way too long. So running only a very few iterations, and then
repeating only on the subset of tests that look "odd" seems to be
working pretty well for me. That's a manual process with a step that
involved subjectively identifying the "odd" cases that should be
executed more. Ideas for automating this process better would also be
appreciated.

-Carl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.freedesktop.org/archives/cairo/attachments/20061214/15e62f17/attachment.pgp