[Intel-gfx] [PATCH i-g-t] lib/igt_core.c: Expand --run-subtest functionality.

[Ville Syrjälä]

On Wed, Jan 27, 2016 at 01:30:01PM +0000, Morton, Derek J wrote:
> >
> >
> >-----Original Message-----
> >From: Ville Syrjälä [mailto:ville.syrjala at linux.intel.com] 
> >Sent: Wednesday, January 27, 2016 12:33 PM
> >To: Morton, Derek J
> >Cc: intel-gfx at lists.freedesktop.org
> >Subject: Re: [Intel-gfx] [PATCH i-g-t] lib/igt_core.c: Expand --run-subtest functionality.
> >
> >On Wed, Jan 27, 2016 at 10:05:56AM +0000, Derek Morton wrote:
> >> Added support for specifying arbitary lists of subtests to run or to 
> >> exclude from being run by using : or ^ as a seperator.
> >> 
> >> :subtest1:subtest2: Will run subtest1 and subtest2 ^subtest1^subtest2^ 
> >> will run all subtests except subtest1 and subtest2
> >
> >Hmm. Getting a bit complicated perhaps. Would it be simpler to just allow specifying the --r option multiple times? So we'd start with the full list of subtests, and each --r option would filter the list in some way?
> 
> By -r I assume you mean --run-subtest? Running a test with -r <subtest> just prints the usage options at the moment.

--r not -r ;)

> Allowing it to be added multiple times could be a way of building up a list of subtests to run, but a completely new command line option would need to then be added (--skip-subtest?) to allow building up a list of subtests to be skipped.

Or could be something like --r !subtest-name

> With my change as is, it allows a string to be passed to the test which details which subtests should be run. If a new parameter such as --skip-subtest is added then it would require knowledge of whether the string contains subtests to run or not to run. That would complicate any scripts that use this to automate testing.
> Allowing --run-subtest (and --skip-subtest) to be specified multiple times will also complicate the code in igt_core.c. Currently there is a simple call to strdup(). If --run-subtest can be specified multiple times the code will have to deal with concatenating strings and any memory reallocation that needs. Also how to deal with the possibility of multiple wildcard expressions?

Hmm. I suppoose my original idea of start with full list and filter
stuff out doesn't work entirely well. Eg. --r foo --r bar would run
nothing. So I guess the option would have to be able to add to the
list as well.

So I guess we could make it so that '!' removes the specified test(s),
no-'!' adds them, and if this is the first --r option and there's no
'!' we'd clear the list entirely before adding the specified test(s)
to it.

> 
> I think that will just end up more complicated than the simple separated list solution this patch introduces.

I suppose. But it would avoid adding a new language which can look
a bit like a weird regexp but isn't.

Maybe if you just use ',' to separate the subtests specifications,
and '!' to specify the not condition things would look a bit more
standardish.

> 
> //Derek
> 
> >
> >> 
> >> Any subtest string not starting : or ^ is treated as a normal wildcard 
> >> expression.
> >> 
> >> This is required mainly on android to exclude subtests that test 
> >> features that do not exist in the android driver while still being 
> >> able to run other subtests in the binary when a wildcard expression is 
> >> insufficient.
> >> 
> >> Signed-off-by: Derek Morton <derek.j.morton at intel.com>
> >> ---
> >>  lib/igt_core.c | 42 ++++++++++++++++++++++++++++++++++++++++--
> >>  1 file changed, 40 insertions(+), 2 deletions(-)
> >> 
> >> diff --git a/lib/igt_core.c b/lib/igt_core.c index 6b69bb7..b9e7470 
> >> 100644
> >> --- a/lib/igt_core.c
> >> +++ b/lib/igt_core.c
> >> @@ -207,7 +207,15 @@
> >>   * To do that obtain the lists of subtests with "--list-subtests", which can be
> >>   * run as non-root and doesn't require the i915 driver to be loaded (or any
> >>   * intel gpu to be present). Then individual subtests can be run with
> >> - * "--run-subtest". Usage help for tests with subtests can be 
> >> obtained with the
> >> + * "--run-subtest". --run-subtest accepts wildcard characters. A list 
> >> + of
> >> + * subtests to run may be specified by using : as a seperator. A list 
> >> + of
> >> + * subtests to exclude may be specified using ^ as a seperator.
> >> + *
> >> + * - --run-subtest basic* will run all subtests starting basic.
> >> + * - --run-subtest :subtest1:subtest2: will run only subtest1 and 
> >> + subtest2
> >> + * - --run-subtest ^subtest1^subtest2^ will run all except subtest1 
> >> + and subtest2
> >> + *
> >> + * Usage help for tests with subtests can be obtained with the
> >>   * "--help" command line option.
> >>   */
> >>  
> >> @@ -786,6 +794,35 @@ void igt_simple_init_parse_opts(int *argc, char **argv,
> >>  		    extra_opt_handler, handler_data);  }
> >>  
> >> +static bool check_testlist(const char *subtest_name) {
> >> +	char *p;
> >> +
> >> +	/* Run subtests in list
> >> +	 * Look for subtest_name in list of form :subtest1:subtest2:subtest3:
> >> +	 * return true if found.
> >> +	 */
> >> +	if (run_single_subtest[0] == ':') {
> >> +		p = strstr(run_single_subtest, subtest_name);
> >> +		if ((p) && (*(p-1) == ':') && (*(p+strlen(subtest_name)) == ':' ))
> >> +			return true;
> >> +	}
> >> +	/* Run subtests not in list
> >> +	 * Look for subtest_name in list of form ^test1^subtest2^subtest3^
> >> +	 * return true if not found.
> >> +	 */
> >> +	else if (run_single_subtest[0] == '^') {
> >> +		p = strstr(run_single_subtest, subtest_name);
> >> +		if (!((p) && (*(p-1) == '^') && (*(p+strlen(subtest_name)) == '^' )))
> >> +			return true;
> >> +	}
> >> +	/* Run subtests that match shell wildcard */
> >> +	else if (fnmatch(run_single_subtest, subtest_name, 0) == 0)
> >> +		return true;
> >> +
> >> +	return false;
> >> +}
> >> +
> >>  /*
> >>   * Note: Testcases which use these helpers MUST NOT output anything to stdout
> >>   * outside of places protected by igt_run_subtest checks - the piglit 
> >> @@ -814,7 +851,8 @@ bool __igt_run_subtest(const char *subtest_name)
> >>  	}
> >>  
> >>  	if (run_single_subtest) {
> >> -		if (fnmatch(run_single_subtest, subtest_name, 0) != 0)
> >> +
> >> +		if (check_testlist(subtest_name) == false)
> >>  			return false;
> >>  		else
> >>  			run_single_subtest_found = true;
> >> --
> >> 1.9.1
> >> 
> >> _______________________________________________
> >> Intel-gfx mailing list
> >> Intel-gfx at lists.freedesktop.org
> >> http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> >
> >--
> >Ville Syrjälä
> >Intel OTC
> >

-- 
Ville Syrjälä
Intel OTC