[Intel-gfx] [PATCH i-g-t] kms_atomic_transition: Add subtest time limit/randomize plane, pipe combinations

[Chris Wilson]

Quoting Imre Deak (2017-11-01 09:56:22)
> On Tue, Oct 31, 2017 at 10:23:25PM +0000, Chris Wilson wrote:
> > Quoting Imre Deak (2017-10-31 13:44:47)
> > > Doing modeset on internal panels may have a considerable overhead due to
> > > the panel specific power sequencing delays. To avoid long test runtimes
> > > limit the runtime of each subtest. Randomize the plane/pipe combinations
> > > to preserve the test coverage on such panels at least over multiple test
> > > runs.
> > > 
> > > Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=103334
> > > Cc: Maarten Lankhorst <maarten.lankhorst at linux.intel.com>
> > > Signed-off-by: Imre Deak <imre.deak at intel.com>
> > > ---
> > >  tests/kms_atomic_transition.c | 175 ++++++++++++++++++++++++++++++++++++------
> > >  1 file changed, 150 insertions(+), 25 deletions(-)
> > > 
> > > diff --git a/tests/kms_atomic_transition.c b/tests/kms_atomic_transition.c
> > > index 4c295125..ac67fc3a 100644
> > > --- a/tests/kms_atomic_transition.c
> > > +++ b/tests/kms_atomic_transition.c
> > > @@ -39,6 +39,14 @@
> > >  #define DRM_CAP_CURSOR_HEIGHT 0x9
> > >  #endif
> > >  
> > > +#define MAX_SUBTEST_DURATION_NS (20ULL * NSEC_PER_SEC)
> > > +
> > > +struct test_config {
> > > +       igt_display_t *display;
> > > +       bool user_seed;
> > > +       int seed;
> > > +};
> > > +
> > >  struct plane_parms {
> > >         struct igt_fb *fb;
> > >         uint32_t width, height;
> > > @@ -401,6 +409,28 @@ static void wait_for_transition(igt_display_t *display, enum pipe pipe, bool non
> > >         }
> > >  }
> > >  
> > > +/* Copied from https://benpfaff.org/writings/clc/shuffle.html */
> > > +static void shuffle_array(uint32_t *array, int size, int seed)
> > > +{
> > > +       int i;
> > > +
> > > +       for (i = 0; i < size; i++) {
> > > +               int j = i + rand() / (RAND_MAX / (size - i) + 1);
> > > +
> > > +               igt_swap(array[i], array[j]);
> > > +       }
> > > +}
> > 
> > igt_permute_array()
> 
> Thanks, will use that instead.
> 
> > Not saying anything, but I was told using CI for stochastic coverage was
> > a flat no...
> 
> Ok, but it would only be the case for slow panels where the alternative
> is not to run the test at all. And is this a problem if we can replay a
> failing case with --seed?

If the purpose of CI is purely regression testing and not exploratory
debugging, each PW run must be with the same seed as the CI_DRM run.

The seed must be clearly displayed in the results so that when comparing
CI_DRM runs the flip-flop can be traced to the change in seed.
-Chris