[Intel-gfx] [PATCH igt] intel-ci: CI is missing any wait tests in BAT

[Chris Wilson]

On Fri, Oct 28, 2016 at 01:46:51PM +0300, Petri Latvala wrote:
> On Thu, Oct 13, 2016 at 03:00:43PM +0100, Chris Wilson wrote:
> > Add the two basic gem_wait tests to the fast list, together they take a
> > total of 1s (when correctly functioning).
> > 
> > Signed-off-by: Chris Wilson <chris at chris-wilson.co.uk>
> > ---
> >  tests/intel-ci/fast-feedback.testlist | 2 ++
> >  1 file changed, 2 insertions(+)
> > 
> > diff --git a/tests/intel-ci/fast-feedback.testlist b/tests/intel-ci/fast-feedback.testlist
> > index f59ec98..853b911 100644
> > --- a/tests/intel-ci/fast-feedback.testlist
> > +++ b/tests/intel-ci/fast-feedback.testlist
> > @@ -126,6 +126,8 @@ igt at gem_sync@basic-store-each
> >  igt at gem_tiled_blits@basic
> >  igt at gem_tiled_fence_blits@basic
> >  igt at gem_tiled_pread_basic
> > +igt at gem_wait@basic-busy-all
> > +igt at gem_wait@basic-wait-all
> >  igt at gem_workarounds@basic-read
> >  igt at gvt_basic@invalid-placeholder-test
> >  igt at kms_addfb_basic@addfb25-bad-modifier
> > -- 
> > 2.9.3
> 
> 
> In the future we will test new tests in the CI system. Unfortunately
> the technology isn't there yet (tm) so this had to be tested
> manually. Sorry for the delay.
> 
> For the record, the testing requirements for incoming new tests is
> converging to:
> 
> - Tests must be stable. 1000 repeated runs with the new tests alone
>   must give the same results. 10 repeated runs of modified BAT set
>   must give the same results (excluding known issues in other tests).
> - Tests must be fast. Full modified BAT run must not exceed the time
>   budget.
> - Tests must pass on current kernels, on some platform.

An idea I had in passing... Remove any test from BAT that has not failed
in the last 180days of trybot and replace it with a current failing
test. BAT is useless if it is not detecting the right class of bugs,
those that we typically make mistakes in (i.e. we should optimise BAT
for coverage of coding errors). Plus we also look at BAT results pretty
frequency and failing tests get fixed faster due to the greater nagging
and faster response time of the automated system. (If by the end of the
week we haven't fixed the new failing cases, we are slacking!) Some
caveats apply, we definitely want to ensure coverage of ABI and the
tests must still be selected to fit within a time budget (I hopeful that
an expert system backed up by coverage analysis would be extremely
invaluable in refining our tests sets). Filtering of trybot results
would have to be done, probably with a voting system: select the most
frequently failed test cases, and then discount failures in other tests
in the same runs (based on the assumption that the same bug was seem by
the different tests, given a large enough sample size should ensure good
coverage).

Just an idea based around tests that always pass are boring (and a waste
of time), but tests we occasionally fail are more sensitive to our errors.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre