[Mesa-dev] [PATCH] i965: Don't check for draw-time errors that cannot occur in core profile

[Ilia Mirkin]

On Tue, Sep 1, 2015 at 12:15 PM, Ian Romanick <idr at freedesktop.org> wrote:
> For a bunch of the small changes, I don't care too much what the
> difference is.  I just want to know whether after is better than before.

And that gets back to my comment that you can't *measure* the impact
of a change. Not with something where the outcome is a random
variable. It can't be done.

All you can do is answer the question "is X's mean more than N higher
than Y's mean". And you change the number of trials in an experiment
depending on N. (There's also more advanced concepts like 'power' and
whatnot, I've done just fine without fully understanding them, I
suspect you can too.)

As an aside, increasing the number of trials until you get a
significant result is a great way to arrive at incorrect decisions,
due to the multi-look problem (95% CI means 1/20 gives you bad
results). The proper way is to decide beforehand "I care about changes
>0.1%, which means I need to run 5000 trial runs" (based on the
assumption that 50 runs gets you 1%). After doing the 5k runs, your CI
width should be ~0.1% and you should then be able to see if the delta
in means is higher or lower than that. If it's higher, then you've
detected a significant change. If it's not, that btw doesn't mean "no
change", just not statistically significant. There's also a procedure
for the null hypothesis (i.e. is a change's impact <1%) which is
basically the same thing but involves doing a few more runs (like 50%
more? I forget the details).

Anyways, I'm sure I've bored everyone to death with these pedantic
explanations, but IME statistics is one of the most misunderstood
areas of math, especially among us engineers.

  -ilia