[Intel-gfx] [RFC i-g-t v2] igt: Test tagging support

[Martin Peres]

On 21/07/17 15:17, Tvrtko Ursulin wrote:
> 
> On 21/07/2017 13:02, Martin Peres wrote:
>> On 21/07/17 14:27, Chris Wilson wrote:
>>> Quoting Tvrtko Ursulin (2017-07-21 12:08:00)
>>>>
>>>> On 21/07/2017 11:36, Chris Wilson wrote:
>>>>> Quoting Tvrtko Ursulin (2017-07-21 11:20:05)
>>>>>> From: Tvrtko Ursulin <tvrtko.ursulin at intel.com>
>>>>
>>>> [snip]
>>>>
>>>>>> --- a/tests/gem_concurrent_all.c
>>>>>> +++ b/tests/gem_concurrent_all.c
>>>>>> @@ -1492,47 +1492,47 @@ run_mode(const char *prefix,
>>>>>>                           igt_subtest_group  {
>>>>>>                                   igt_fixture p->require();
>>>>>> - igt_subtest_f("%s-%s-%s-sanitycheck0%s%s", prefix, mode->name, 
>>>>>> p->prefix, suffix, h->suffix) {
>>>>>> +                               igt_gem_stress_subtest_f("", 
>>>>>> "%s-%s-%s-sanitycheck0%s%s", prefix, mode->name, p->prefix, 
>>>>>> suffix, h->suffix) {
>>>>>
>>>>> They are not all stress tests. So you want to be able to build the 
>>>>> tags
>>>>> dynamically... Similarly they offer different types of "stress", you
>>>>> probably don't want to lump the hang tests in amongst thes plain
>>>>> concurrency tests, and you probably want the swapping tests separated
>>>>> etc. Stress is missing the point.
>>>>
>>>> Dynamic tags are doable. If you just wanted to include "stress"
>>>> dynamically current RFC can already do that.
>>>>
>>>>          igt_gem_subtest_f(is_stress ? "stress" : "", name, ...)
>>>>
>>>> If you wanted a dynamic set of multiple tags that could be added as 
>>>> well
>>>> I guess. Like a flag based control of "stress", "swapping", "hang",
>>>> "basic", or something. How nice or ugly API depends on the actual
>>>> requirements.
>>>
>>> hang, swap, shrink, gtt, wc, cpu, pwrite, pread, contexts, fds, prime,
>>> dmabuf and many more when you start looking for the complete set of
>>> tags/keywords/categories.
>>
>> This is why I would rather use the execution time of tests as a way to 
>> tag the tests. What I want is to have a couple of options that brings 
>> me the best coverage in a certain amount of time:
>>   - BAT: 70% coverage in 10 minutes
>>   - FULL: 95% coverage in 6 hours
>>   - Stress: 99% coverage in 1 year
>>
>> What do the tags hang, swap, shrink, gtt, etc.. are supposed to bring 
>> to me?
> 
> Two things. First is for the "full" run, CI, developers, whoever:
> 
> run-tests --exclude stress
> 
> And then for the developers ("Making IGT runnable by CI and 
> developers"!), for example after some refactoring in specific areas of 
> the code base:
> 
> run-tests --include gem --include pread
> run-tests --include kms --include flip --exclude hang
> 
> Examples only...

I see, and I like that! So tests are annotated with many tags, that 
makes sense :)