[Intel-gfx] [RFC i-g-t v2] igt: Test tagging support

[Martin Peres]

On 21/07/17 14:27, Chris Wilson wrote:
> Quoting Tvrtko Ursulin (2017-07-21 12:08:00)
>>
>> On 21/07/2017 11:36, Chris Wilson wrote:
>>> Quoting Tvrtko Ursulin (2017-07-21 11:20:05)
>>>> From: Tvrtko Ursulin <tvrtko.ursulin at intel.com>
>>
>> [snip]
>>
>>>> --- a/tests/gem_concurrent_all.c
>>>> +++ b/tests/gem_concurrent_all.c
>>>> @@ -1492,47 +1492,47 @@ run_mode(const char *prefix,
>>>>                           igt_subtest_group  {
>>>>                                   igt_fixture p->require();
>>>>    
>>>> -                               igt_subtest_f("%s-%s-%s-sanitycheck0%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
>>>> +                               igt_gem_stress_subtest_f("", "%s-%s-%s-sanitycheck0%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
>>>
>>> They are not all stress tests. So you want to be able to build the tags
>>> dynamically... Similarly they offer different types of "stress", you
>>> probably don't want to lump the hang tests in amongst thes plain
>>> concurrency tests, and you probably want the swapping tests separated
>>> etc. Stress is missing the point.
>>
>> Dynamic tags are doable. If you just wanted to include "stress"
>> dynamically current RFC can already do that.
>>
>>          igt_gem_subtest_f(is_stress ? "stress" : "", name, ...)
>>
>> If you wanted a dynamic set of multiple tags that could be added as well
>> I guess. Like a flag based control of "stress", "swapping", "hang",
>> "basic", or something. How nice or ugly API depends on the actual
>> requirements.
> 
> hang, swap, shrink, gtt, wc, cpu, pwrite, pread, contexts, fds, prime,
> dmabuf and many more when you start looking for the complete set of
> tags/keywords/categories.

This is why I would rather use the execution time of tests as a way to 
tag the tests. What I want is to have a couple of options that brings me 
the best coverage in a certain amount of time:
  - BAT: 70% coverage in 10 minutes
  - FULL: 95% coverage in 6 hours
  - Stress: 99% coverage in 1 year

What do the tags hang, swap, shrink, gtt, etc.. are supposed to bring to me?

Martin