[Intel-gfx] [PATCH i-g-t] lib/igt_core.c: Expand --run-subtest functionality.

Ville Syrjälä ville.syrjala at linux.intel.com
Wed Jan 27 07:36:09 PST 2016 

On Wed, Jan 27, 2016 at 03:02:15PM +0000, Morton, Derek J wrote:
> >
> >
> >-----Original Message-----
> >From: Ville Syrjälä [mailto:ville.syrjala at linux.intel.com] 
> >Sent: Wednesday, January 27, 2016 2:31 PM
> >To: Morton, Derek J
> >Cc: intel-gfx at lists.freedesktop.org
> >Subject: Re: [Intel-gfx] [PATCH i-g-t] lib/igt_core.c: Expand --run-subtest functionality.
> >
> >On Wed, Jan 27, 2016 at 01:30:01PM +0000, Morton, Derek J wrote:
> >> >
> >> >
> >> >-----Original Message-----
> >> >From: Ville Syrjälä [mailto:ville.syrjala at linux.intel.com]
> >> >Sent: Wednesday, January 27, 2016 12:33 PM
> >> >To: Morton, Derek J
> >> >Cc: intel-gfx at lists.freedesktop.org
> >> >Subject: Re: [Intel-gfx] [PATCH i-g-t] lib/igt_core.c: Expand --run-subtest functionality.
> >> >
> >> >On Wed, Jan 27, 2016 at 10:05:56AM +0000, Derek Morton wrote:
> >> >> Added support for specifying arbitary lists of subtests to run or 
> >> >> to exclude from being run by using : or ^ as a seperator.
> >> >> 
> >> >> :subtest1:subtest2: Will run subtest1 and subtest2 
> >> >> ^subtest1^subtest2^ will run all subtests except subtest1 and 
> >> >> subtest2
> >> >
> >> >Hmm. Getting a bit complicated perhaps. Would it be simpler to just allow specifying the --r option multiple times? So we'd start with the full list of subtests, and each --r option would filter the list in some way?
> >> 
> >> By -r I assume you mean --run-subtest? Running a test with -r <subtest> just prints the usage options at the moment.
> >
> >--r not -r ;)
> >
> >> Allowing it to be added multiple times could be a way of building up a list of subtests to run, but a completely new command line option would need to then be added (--skip-subtest?) to allow building up a list of subtests to be skipped.
> >
> >Or could be something like --r !subtest-name
> >
> >> With my change as is, it allows a string to be passed to the test which details which subtests should be run. If a new parameter such as --skip-subtest is added then it would require knowledge of whether the string contains subtests to run or not to run. That would complicate any scripts that use this to automate testing.
> >> Allowing --run-subtest (and --skip-subtest) to be specified multiple times will also complicate the code in igt_core.c. Currently there is a simple call to strdup(). If --run-subtest can be specified multiple times the code will have to deal with concatenating strings and any memory reallocation that needs. Also how to deal with the possibility of multiple wildcard expressions?
> >
> >Hmm. I suppoose my original idea of start with full list and filter stuff out doesn't work entirely well. Eg. --r foo --r bar would run nothing. So I guess the option would have to be able to add to the list as well.
> >
> >So I guess we could make it so that '!' removes the specified test(s), no-'!' adds them, and if this is the first --r option and there's no '!' we'd clear the list entirely before adding the specified test(s) to it.
> >
> >> 
> >> I think that will just end up more complicated than the simple separated list solution this patch introduces.
> >
> >I suppose. But it would avoid adding a new language which can look a bit like a weird regexp but isn't.
> >
> >Maybe if you just use ',' to separate the subtests specifications, and '!' to specify the not condition things would look a bit more standardish.
> 
> I can't use ! as the shell uses it for some command history substitution. That is why I chose ^. There are very few special character the shell does not mess up :-(.

A bunch of stuff uses ! (eg. test, find, etc.). The shell won't eat it if
there's a space after it, or you could always escape it. getopt doesn't
support it natively however so it could a pain to use the space trick.

> 
> How about the following:
> 
> Comma separated list of subtests to specify a list of subtests to run.
> Prepend ^ to the list to specify a list of subtests to exclude.
> 
> I would then have to use strstr to look for a comma in what is specified to --run-subtests. Is there any risk of a comma in a wildcard expression? I found some code for fnmatch with google and it looks like it is simply ?*[]- which is treated as special characters.
> 
> //Derek 
> 
> >
> >> 
> >> //Derek
> >> 
> >> >
> >> >> 
> >> >> Any subtest string not starting : or ^ is treated as a normal 
> >> >> wildcard expression.
> >> >> 
> >> >> This is required mainly on android to exclude subtests that test 
> >> >> features that do not exist in the android driver while still being 
> >> >> able to run other subtests in the binary when a wildcard expression 
> >> >> is insufficient.
> >> >> 
> >> >> Signed-off-by: Derek Morton <derek.j.morton at intel.com>
> >> >> ---
> >> >>  lib/igt_core.c | 42 ++++++++++++++++++++++++++++++++++++++++--
> >> >>  1 file changed, 40 insertions(+), 2 deletions(-)
> >> >> 
> >> >> diff --git a/lib/igt_core.c b/lib/igt_core.c index 6b69bb7..b9e7470
> >> >> 100644
> >> >> --- a/lib/igt_core.c
> >> >> +++ b/lib/igt_core.c
> >> >> @@ -207,7 +207,15 @@
> >> >>   * To do that obtain the lists of subtests with "--list-subtests", which can be
> >> >>   * run as non-root and doesn't require the i915 driver to be loaded (or any
> >> >>   * intel gpu to be present). Then individual subtests can be run 
> >> >> with
> >> >> - * "--run-subtest". Usage help for tests with subtests can be 
> >> >> obtained with the
> >> >> + * "--run-subtest". --run-subtest accepts wildcard characters. A 
> >> >> + list of
> >> >> + * subtests to run may be specified by using : as a seperator. A 
> >> >> + list of
> >> >> + * subtests to exclude may be specified using ^ as a seperator.
> >> >> + *
> >> >> + * - --run-subtest basic* will run all subtests starting basic.
> >> >> + * - --run-subtest :subtest1:subtest2: will run only subtest1 and
> >> >> + subtest2
> >> >> + * - --run-subtest ^subtest1^subtest2^ will run all except 
> >> >> + subtest1 and subtest2
> >> >> + *
> >> >> + * Usage help for tests with subtests can be obtained with the
> >> >>   * "--help" command line option.
> >> >>   */
> >> >>  
> >> >> @@ -786,6 +794,35 @@ void igt_simple_init_parse_opts(int *argc, char **argv,
> >> >>  		    extra_opt_handler, handler_data);  }
> >> >>  
> >> >> +static bool check_testlist(const char *subtest_name) {
> >> >> +	char *p;
> >> >> +
> >> >> +	/* Run subtests in list
> >> >> +	 * Look for subtest_name in list of form :subtest1:subtest2:subtest3:
> >> >> +	 * return true if found.
> >> >> +	 */
> >> >> +	if (run_single_subtest[0] == ':') {
> >> >> +		p = strstr(run_single_subtest, subtest_name);
> >> >> +		if ((p) && (*(p-1) == ':') && (*(p+strlen(subtest_name)) == ':' ))
> >> >> +			return true;
> >> >> +	}
> >> >> +	/* Run subtests not in list
> >> >> +	 * Look for subtest_name in list of form ^test1^subtest2^subtest3^
> >> >> +	 * return true if not found.
> >> >> +	 */
> >> >> +	else if (run_single_subtest[0] == '^') {
> >> >> +		p = strstr(run_single_subtest, subtest_name);
> >> >> +		if (!((p) && (*(p-1) == '^') && (*(p+strlen(subtest_name)) == '^' )))
> >> >> +			return true;
> >> >> +	}
> >> >> +	/* Run subtests that match shell wildcard */
> >> >> +	else if (fnmatch(run_single_subtest, subtest_name, 0) == 0)
> >> >> +		return true;
> >> >> +
> >> >> +	return false;
> >> >> +}
> >> >> +
> >> >>  /*
> >> >>   * Note: Testcases which use these helpers MUST NOT output anything to stdout
> >> >>   * outside of places protected by igt_run_subtest checks - the 
> >> >> piglit @@ -814,7 +851,8 @@ bool __igt_run_subtest(const char *subtest_name)
> >> >>  	}
> >> >>  
> >> >>  	if (run_single_subtest) {
> >> >> -		if (fnmatch(run_single_subtest, subtest_name, 0) != 0)
> >> >> +
> >> >> +		if (check_testlist(subtest_name) == false)
> >> >>  			return false;
> >> >>  		else
> >> >>  			run_single_subtest_found = true;
> >> >> --
> >> >> 1.9.1
> >> >> 
> >> >> _______________________________________________
> >> >> Intel-gfx mailing list
> >> >> Intel-gfx at lists.freedesktop.org
> >> >> http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> >> >
> >> >--
> >> >Ville Syrjälä
> >> >Intel OTC
> >> >
> >
> >--
> >Ville Syrjälä
> >Intel OTC
> >

-- 
Ville Syrjälä
Intel OTC

More information about the Intel-gfx mailing list