[igt-dev] [PATCH i-g-t] lib: Update selftests to use dynamic subtests

[Sarvela, Tomi P]

> From: Latvala, Petri <petri.latvala at intel.com>
> 
> On Fri, Feb 14, 2020 at 09:48:29PM +0000, Chris Wilson wrote:
> > Use the new igt_subtest_with_dynamic to nicely group the dynamic
> > subtests together.
> >
> > Signed-off-by: Chris Wilson <chris at chris-wilson.co.uk>
> > Cc: Petri Latvala <petri.latvala at intel.com>
> > Cc: Tomi Sarvela <tomi.p.sarvela at intel.com>
> > ---
> >  lib/igt_kmod.c | 23 +++++++++++++++++++----
> >  1 file changed, 19 insertions(+), 4 deletions(-)
> >
> > diff --git a/lib/igt_kmod.c b/lib/igt_kmod.c
> > index 05019c24c..e701545d3 100644
> > --- a/lib/igt_kmod.c
> > +++ b/lib/igt_kmod.c
> > @@ -21,6 +21,7 @@
> >   * IN THE SOFTWARE.
> >   */
> >
> > +#include <ctype.h>
> >  #include <signal.h>
> >  #include <errno.h>
> >
> > @@ -602,6 +603,18 @@ void igt_kselftest_fini(struct igt_kselftest *tst)
> >  	kmod_module_unref(tst->kmod);
> >  }
> >
> > +static const char *unfilter(const char *filter, const char *name)
> > +{
> > +	if (!filter)
> > +		return name;
> > +
> > +	name += strlen(filter);
> > +	if (!isalpha(*name))
> > +		name++;
> > +
> > +	return name;
> > +}
> > +
> >  void igt_kselftests(const char *module_name,
> >  		    const char *options,
> >  		    const char *result,
> > @@ -618,10 +631,12 @@ void igt_kselftests(const char *module_name,
> >  		igt_require(igt_kselftest_begin(&tst) == 0);
> >
> >  	igt_kselftest_get_tests(tst.kmod, filter, &tests);
> > -	igt_list_for_each_entry_safe(tl, tn, &tests, link) {
> > -		igt_subtest_f("%s", tl->name)
> > -			igt_kselftest_execute(&tst, tl,
> options, result);
> > -		free(tl);
> > +	igt_subtest_with_dynamic(filter ?: "all") {
> > +		igt_list_for_each_entry_safe(tl, tn, &tests, link)
> {
> > +			igt_dynamic_f("%s",
> unfilter(filter, tl->name))
> > +
> 	igt_kselftest_execute(&tst, tl, options, result);
> > +			free(tl);
> > +		}
> >  	}
> 
> 
> Thanks for starting this, exactly what dynamic subtests are for.
> 
> Pre-acked-by: Petri Latvala <petri.latvala at intel.com>
> 
> CI side should be made ready before landing this. With kernel
> selftests being now launched from statically enumerable subtests,
> selftests can be added to fast-feedback.testlist and un-blacklisted
> from shards where applicable.

Acked-by: Tomi Sarvela <tomi.p.sarvela at intel.com>

This is good: the discovery of sub-selftests will be done by the tests, so
we can drop the selftest.testlist and use common blacklist where applicable.

> Tomi, what do we run (and what do we want to) in BAT now? Objections
> to running selftests in shards mixed in with other tests?

BAT is running fast-feedback.testlist and after that from selftests:
igt at i915_selftest@live_*
igt at dmabuf@*

In other words, currently blacklisted selftests in BAT are:
igt at i915_selftest@{mock,perf}_*
igt at drm_mm@*
igt at kms_selftest@*

I don't mind selftests being shuffled through all the other tests, we have
the checks in place for taint/lockdep to recognize hopeless driver states
to force reboot.

> Runtime is a concern if all (FSVO all) selftests are launched with one
> entry in a testlist. What is the total runtime on a random platform
> for, say, i915_selftest subtests (post patch)? We might need to split
> more than just live/mock.

Runtimes for last functional selftest shard are from CI_DRM_7925:
tgl 186s
hsw 762s
glk 223s
icl 177s
kbl 199s
apl 284s
snb 112s
skl 763s

The obvious outliers are SKL and HSW:
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7925/shard-skl10/igt_runner35.txt 
https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7925/shard-hsw5/igt_runner35.txt 

HSW due to timeout in live_gtt which wasted 10 minutes, SKL due to generic slowness
and in addition couple of drm_mm tests taking > 100s per subtest.
 
Tomi