[Intel-gfx] [PATCH i-g-t v2] lib/igt_core.c: Expand --run-subtest functionality.

[Morton, Derek J]

Hi,

I have spoken to Dave. I do not want to implement multiple --run-subtest as it is not something I require and I do not think it is worth adding unless someone has a specific need for it. However I understand that whatever is done should not make that harder in the future to add multiple --run-subtest and so the --run-subtest should always be additive. The issue seems to be how to specify "add all subtests except ..." in a 'standard' way. The fnmatch() used currently only supports *?[] so there is no way of specifying 'not'. What would be the feeling if I was to replace fnmatch() with support for --run-subtest supporting proper regex expressions by including regex.h and using regexec()?

As far as I can see the impact of that would be a change of behaviour of * and ? in the expression which might not be desirable if the wildcard expressions are in use. As an alternative I could add a new parameter --run-subtest-regex which allows the list of subtests to be run to be specified with a regex.

Would that be acceptable?

//Derek

>
>
>-----Original Message-----
>From: Gordon, David S 
>Sent: Thursday, January 28, 2016 12:58 PM
>To: Morton, Derek J; intel-gfx at lists.freedesktop.org
>Cc: daniel.vetter at ffwll.ch
>Subject: Re: [Intel-gfx] [PATCH i-g-t v2] lib/igt_core.c: Expand --run-subtest functionality.
>
>On 28/01/16 11:05, Derek Morton wrote:
>> Added support for specifying arbitary lists of subtests to run, or to 
>> exclude from being run if prefixed by ^ or !.
>>
>> subtest1,subtest2 Will run subtest1 and subtest2
>> ^subtest1,subtest2 or !subtest1,subtest2 will run all subtests except
>> subtest1 and subtest2.
>>
>> Any subtest string not starting ! or ^ and not containing a comma is 
>> treated as a normal wildcard expression.
>>
>> This is required mainly on android to exclude subtests that test 
>> features that do not exist in the android driver while still being 
>> able to run other subtests in the binary when a wildcard expression is 
>> insufficient.
>>
>> v2: Use comma as list separator (Ville Syrjala) support both ^ and ! 
>> as not operators (Dave Gordon)
>>
>> Signed-off-by: Derek Morton <derek.j.morton at intel.com>
>> ---
>>   lib/igt_core.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++--
>>   1 file changed, 46 insertions(+), 2 deletions(-)
>>
>> diff --git a/lib/igt_core.c b/lib/igt_core.c index 6b69bb7..5d243af 
>> 100644
>> --- a/lib/igt_core.c
>> +++ b/lib/igt_core.c
>> @@ -207,7 +207,15 @@
>>    * To do that obtain the lists of subtests with "--list-subtests", which can be
>>    * run as non-root and doesn't require the i915 driver to be loaded (or any
>>    * intel gpu to be present). Then individual subtests can be run 
>> with
>> - * "--run-subtest". Usage help for tests with subtests can be 
>> obtained with the
>> + * "--run-subtest". --run-subtest accepts wildcard characters. A list 
>> + of
>> + * subtests to run may be specified by using ',' as a separator.
>> + * A prefix of ^ or ! may be added to invert the logic, e.g. run all tests except...
>> + *
>> + * - --run-subtest basic* will run all subtests starting basic.
>> + * - --run-subtest subtest1,subtest2 will run only subtest1 and 
>> + subtest2
>> + * - --run-subtest ^subtest1,subtest2 will run all those except 
>> + subtest1 and subtest2
>> + *
>> + * Usage help for tests with subtests can be obtained with the
>>    * "--help" command line option.
>>    */
>
>This is the syntax option 1 from my previous mail (but not allowing multiple --run-subtest options?), which I think is a bit less powerful and a bit less desirable because of the perhaps unexpected precedence of OR above NOT (^sub1,sub2 => NOT(sub1 OR sub2).
>
>Do you think the advantages of option 2 (full generality, can express any boolean, expected precedence of operators) would make it worth the extra effort?
>
>Whichever we pick now, it will be set in stone as it will be very inconvenient to edit every script using the feature once it's been in use for a while!
>
>Another idea: have you considered bash(1) extglob syntax? See the manpage for bash, section "Pattern Matching". It's quite powerful, familiar to least to script writers, and easy to try out in bash:
>
>$ : enable extglob mode
>$ shopt -s extglob
>$ : create an array of test names
>$ declare -a tests=( starting phase1 phase2 wibble phase3 wobble done ) $ : see what matches a pattern:
>$ function match() { for w in "${tests[@]}" ; do [[ "$w" == $1 ]] && echo "$w" ; done ; } $ match '*'
>starting
>phase1
>phase2
>wibble
>phase3
>wobble
>done
>$ match 'phase*'
>phase1
>phase2
>phase3
>$ match 'w?bble'
>wibble
>wobble
>$ match '[!pw]*'
>starting
>done
>$ match '@(starting|done)'
>starting
>done
>$ match '!(starting|done)'
>phase1
>phase2
>wibble
>phase3
>wobble
>$ match '!(phase*)'
>starting
>wibble
>wobble
>done
>$ match '!(phase*|w?bble)'
>starting
>done
>$ : etc, etc, ad lib
>
>.Dave.
>
>