[igt-dev] [PATCH i-g-t 6/9] tests/perf_pmu: PMU enable race test

[Chris Wilson]

Quoting Tvrtko Ursulin (2018-02-05 10:40:01)
> 
> On 05/02/2018 10:16, Chris Wilson wrote:
> > Quoting Tvrtko Ursulin (2018-02-05 10:07:02)
> >> I can wrap it in a 10x loop but until the PMU is fixed that means 10x
> >> longer runtime as well which I wanted to avoid, and instead count on
> >> execution in numbers across the shards.
> > 
> > Otoh, I want a test that we can reliably run locally to determine if we
> > have a problem or not. A difficult balance. What should we aim for
> > flip-floppy tests, >50% error rate? >10%? Certainly has to be >1% or we
> > will struggle to notice the pattern. My finger in the air would say >33%.
> 
> I don't know how we do that since it will depend on so many factors. On 
> my SKL GT2 and my kernel config, it is maybe around 10%. So if we cannot 
> reach >33% what do we, don't add the test?

I'm flexible, though if we can't force it to hit the race even when we
know what we're looking for, that doesn't bode well :) I think if we
knowingly add a flip-flop, we at least try to capture what the
reliability of the test is for a given setup in the changelog and
comments. Just so we know what we are looking at in 6 months time.
-Chris