[Intel-gfx] [PATCH i-g-t] kms_atomic_transition: Add subtest time limit/randomize plane, pipe combinations

[Imre Deak]

On Wed, Nov 01, 2017 at 01:08:47PM +0200, Imre Deak wrote:
> On Wed, Nov 01, 2017 at 10:48:50AM +0000, Chris Wilson wrote:
> > Quoting Imre Deak (2017-11-01 09:56:22)
> > > On Tue, Oct 31, 2017 at 10:23:25PM +0000, Chris Wilson wrote:
> > > > Quoting Imre Deak (2017-10-31 13:44:47)
> > > > > Doing modeset on internal panels may have a considerable overhead due to
> > > > > the panel specific power sequencing delays. To avoid long test runtimes
> > > > > limit the runtime of each subtest. Randomize the plane/pipe combinations
> > > > > to preserve the test coverage on such panels at least over multiple test
> > > > > runs.
> > > > > 
> > > > > Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=103334
> > > > > Cc: Maarten Lankhorst <maarten.lankhorst at linux.intel.com>
> > > > > Signed-off-by: Imre Deak <imre.deak at intel.com>
> > > > > ---
> > > > >  tests/kms_atomic_transition.c | 175 ++++++++++++++++++++++++++++++++++++------
> > > > >  1 file changed, 150 insertions(+), 25 deletions(-)
> > > > > 
> > > > > diff --git a/tests/kms_atomic_transition.c b/tests/kms_atomic_transition.c
> > > > > index 4c295125..ac67fc3a 100644
> > > > > --- a/tests/kms_atomic_transition.c
> > > > > +++ b/tests/kms_atomic_transition.c
> > > > > @@ -39,6 +39,14 @@
> > > > >  #define DRM_CAP_CURSOR_HEIGHT 0x9
> > > > >  #endif
> > > > >  
> > > > > +#define MAX_SUBTEST_DURATION_NS (20ULL * NSEC_PER_SEC)
> > > > > +
> > > > > +struct test_config {
> > > > > +       igt_display_t *display;
> > > > > +       bool user_seed;
> > > > > +       int seed;
> > > > > +};
> > > > > +
> > > > >  struct plane_parms {
> > > > >         struct igt_fb *fb;
> > > > >         uint32_t width, height;
> > > > > @@ -401,6 +409,28 @@ static void wait_for_transition(igt_display_t *display, enum pipe pipe, bool non
> > > > >         }
> > > > >  }
> > > > >  
> > > > > +/* Copied from https://benpfaff.org/writings/clc/shuffle.html */
> > > > > +static void shuffle_array(uint32_t *array, int size, int seed)
> > > > > +{
> > > > > +       int i;
> > > > > +
> > > > > +       for (i = 0; i < size; i++) {
> > > > > +               int j = i + rand() / (RAND_MAX / (size - i) + 1);
> > > > > +
> > > > > +               igt_swap(array[i], array[j]);
> > > > > +       }
> > > > > +}
> > > > 
> > > > igt_permute_array()
> > > 
> > > Thanks, will use that instead.
> > > 
> > > > Not saying anything, but I was told using CI for stochastic coverage was
> > > > a flat no...
> > > 
> > > Ok, but it would only be the case for slow panels where the alternative
> > > is not to run the test at all. And is this a problem if we can replay a
> > > failing case with --seed?
> > 
> > If the purpose of CI is purely regression testing and not exploratory
> > debugging, each PW run must be with the same seed as the CI_DRM run.
> > 
> > The seed must be clearly displayed in the results so that when comparing
> > CI_DRM runs the flip-flop can be traced to the change in seed.
> 
> Adding Petri and Tomi. So I guess the question is if we want this in CI
> and if it's a reasonable effort to implement it.

Discussing more with Tomi and Martin, the randomization idea is not so
practical at least for now.

So another approach would be to split out the testing on internal panels
from plane-all-modeset-transition and
plane-all-modeset-transition-fencing. We could have fast and slow
version of these internal-panel tests where the fast one would test all
combinations and the slow one only combinations with either all planes
on/off or only a single plane on. We could then include only the fast
versions in the fast-feedback testlist.

Will follow up with a version doing the above.

--Imre