[Intel-gfx] [RFC i-g-t v2] igt: Test tagging support

[Tvrtko Ursulin]

On 21/07/2017 13:02, Martin Peres wrote:
> On 21/07/17 14:27, Chris Wilson wrote:
>> Quoting Tvrtko Ursulin (2017-07-21 12:08:00)
>>>
>>> On 21/07/2017 11:36, Chris Wilson wrote:
>>>> Quoting Tvrtko Ursulin (2017-07-21 11:20:05)
>>>>> From: Tvrtko Ursulin <tvrtko.ursulin at intel.com>
>>>
>>> [snip]
>>>
>>>>> --- a/tests/gem_concurrent_all.c
>>>>> +++ b/tests/gem_concurrent_all.c
>>>>> @@ -1492,47 +1492,47 @@ run_mode(const char *prefix,
>>>>>                           igt_subtest_group  {
>>>>>                                   igt_fixture p->require();
>>>>> -                               
>>>>> igt_subtest_f("%s-%s-%s-sanitycheck0%s%s", prefix, mode->name, 
>>>>> p->prefix, suffix, h->suffix) {
>>>>> +                               igt_gem_stress_subtest_f("", 
>>>>> "%s-%s-%s-sanitycheck0%s%s", prefix, mode->name, p->prefix, suffix, 
>>>>> h->suffix) {
>>>>
>>>> They are not all stress tests. So you want to be able to build the tags
>>>> dynamically... Similarly they offer different types of "stress", you
>>>> probably don't want to lump the hang tests in amongst thes plain
>>>> concurrency tests, and you probably want the swapping tests separated
>>>> etc. Stress is missing the point.
>>>
>>> Dynamic tags are doable. If you just wanted to include "stress"
>>> dynamically current RFC can already do that.
>>>
>>>          igt_gem_subtest_f(is_stress ? "stress" : "", name, ...)
>>>
>>> If you wanted a dynamic set of multiple tags that could be added as well
>>> I guess. Like a flag based control of "stress", "swapping", "hang",
>>> "basic", or something. How nice or ugly API depends on the actual
>>> requirements.
>>
>> hang, swap, shrink, gtt, wc, cpu, pwrite, pread, contexts, fds, prime,
>> dmabuf and many more when you start looking for the complete set of
>> tags/keywords/categories.
> 
> This is why I would rather use the execution time of tests as a way to 
> tag the tests. What I want is to have a couple of options that brings me 
> the best coverage in a certain amount of time:
>   - BAT: 70% coverage in 10 minutes
>   - FULL: 95% coverage in 6 hours
>   - Stress: 99% coverage in 1 year
> 
> What do the tags hang, swap, shrink, gtt, etc.. are supposed to bring to 
> me?

Two things. First is for the "full" run, CI, developers, whoever:

run-tests --exclude stress

And then for the developers ("Making IGT runnable by CI and 
developers"!), for example after some refactoring in specific areas of 
the code base:

run-tests --include gem --include pread
run-tests --include kms --include flip --exclude hang

Examples only...

Regards,

Tvrtko