[Intel-gfx] [PATCH i-g-t] kms_atomic_transition: Add subtest time limit/randomize plane, pipe combinations

[Imre Deak]

On Wed, Nov 01, 2017 at 10:48:50AM +0000, Chris Wilson wrote:
> Quoting Imre Deak (2017-11-01 09:56:22)
> > On Tue, Oct 31, 2017 at 10:23:25PM +0000, Chris Wilson wrote:
> > > Quoting Imre Deak (2017-10-31 13:44:47)
> > > > Doing modeset on internal panels may have a considerable overhead due to
> > > > the panel specific power sequencing delays. To avoid long test runtimes
> > > > limit the runtime of each subtest. Randomize the plane/pipe combinations
> > > > to preserve the test coverage on such panels at least over multiple test
> > > > runs.
> > > > 
> > > > Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=103334
> > > > Cc: Maarten Lankhorst <maarten.lankhorst at linux.intel.com>
> > > > Signed-off-by: Imre Deak <imre.deak at intel.com>
> > > > ---
> > > >  tests/kms_atomic_transition.c | 175 ++++++++++++++++++++++++++++++++++++------
> > > >  1 file changed, 150 insertions(+), 25 deletions(-)
> > > > 
> > > > diff --git a/tests/kms_atomic_transition.c b/tests/kms_atomic_transition.c
> > > > index 4c295125..ac67fc3a 100644
> > > > --- a/tests/kms_atomic_transition.c
> > > > +++ b/tests/kms_atomic_transition.c
> > > > @@ -39,6 +39,14 @@
> > > >  #define DRM_CAP_CURSOR_HEIGHT 0x9
> > > >  #endif
> > > >  
> > > > +#define MAX_SUBTEST_DURATION_NS (20ULL * NSEC_PER_SEC)
> > > > +
> > > > +struct test_config {
> > > > +       igt_display_t *display;
> > > > +       bool user_seed;
> > > > +       int seed;
> > > > +};
> > > > +
> > > >  struct plane_parms {
> > > >         struct igt_fb *fb;
> > > >         uint32_t width, height;
> > > > @@ -401,6 +409,28 @@ static void wait_for_transition(igt_display_t *display, enum pipe pipe, bool non
> > > >         }
> > > >  }
> > > >  
> > > > +/* Copied from https://benpfaff.org/writings/clc/shuffle.html */
> > > > +static void shuffle_array(uint32_t *array, int size, int seed)
> > > > +{
> > > > +       int i;
> > > > +
> > > > +       for (i = 0; i < size; i++) {
> > > > +               int j = i + rand() / (RAND_MAX / (size - i) + 1);
> > > > +
> > > > +               igt_swap(array[i], array[j]);
> > > > +       }
> > > > +}
> > > 
> > > igt_permute_array()
> > 
> > Thanks, will use that instead.
> > 
> > > Not saying anything, but I was told using CI for stochastic coverage was
> > > a flat no...
> > 
> > Ok, but it would only be the case for slow panels where the alternative
> > is not to run the test at all. And is this a problem if we can replay a
> > failing case with --seed?
> 
> If the purpose of CI is purely regression testing and not exploratory
> debugging, each PW run must be with the same seed as the CI_DRM run.
> 
> The seed must be clearly displayed in the results so that when comparing
> CI_DRM runs the flip-flop can be traced to the change in seed.

Adding Petri and Tomi. So I guess the question is if we want this in CI
and if it's a reasonable effort to implement it.

--Imre