[Intel-gfx] [RFC i-g-t v2] igt: Test tagging support

[Tvrtko Ursulin]

On 21/07/2017 12:27, Chris Wilson wrote:
> Quoting Tvrtko Ursulin (2017-07-21 12:08:00)
>>
>> On 21/07/2017 11:36, Chris Wilson wrote:
>>> Quoting Tvrtko Ursulin (2017-07-21 11:20:05)
>>>> From: Tvrtko Ursulin <tvrtko.ursulin at intel.com>
>>
>> [snip]
>>
>>>> --- a/tests/gem_concurrent_all.c
>>>> +++ b/tests/gem_concurrent_all.c
>>>> @@ -1492,47 +1492,47 @@ run_mode(const char *prefix,
>>>>                           igt_subtest_group  {
>>>>                                   igt_fixture p->require();
>>>>    
>>>> -                               igt_subtest_f("%s-%s-%s-sanitycheck0%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
>>>> +                               igt_gem_stress_subtest_f("", "%s-%s-%s-sanitycheck0%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
>>>
>>> They are not all stress tests. So you want to be able to build the tags
>>> dynamically... Similarly they offer different types of "stress", you
>>> probably don't want to lump the hang tests in amongst thes plain
>>> concurrency tests, and you probably want the swapping tests separated
>>> etc. Stress is missing the point.
>>
>> Dynamic tags are doable. If you just wanted to include "stress"
>> dynamically current RFC can already do that.
>>
>>          igt_gem_subtest_f(is_stress ? "stress" : "", name, ...)
>>
>> If you wanted a dynamic set of multiple tags that could be added as well
>> I guess. Like a flag based control of "stress", "swapping", "hang",
>> "basic", or something. How nice or ugly API depends on the actual
>> requirements.
> 
> hang, swap, shrink, gtt, wc, cpu, pwrite, pread, contexts, fds, prime,
> dmabuf and many more when you start looking for the complete set of
> tags/keywords/categories.

I would like having rich tags as well.

> Currently, we have tags (keywords) embedded into the test/subtest name.
> (One way of looking at it, every test would be a unique combination of
> tags.) Being able to filter tests on those tags is definitely lacking.

I don't think every test is an unique combination of tags. For example 
tagged with "gem pread" we woud have tests doing small reads, large 
reads, overflowing reads etc. I would invent new tags for all these cases.

Anyway, I am not sure what is the high level message here. Do you like 
the general idea but find the implementation lacking, or think something 
completely different would be the easier route to the equally functional 
end result?

I like tags because they separate from the test naming namespace. I used 
the example of "default" which means different things in the current 
test names. But as I said it is just a proposal.

And gem_concurrent_* is probably on of the hardest ones to elegantly 
handle since the name is generated from so many parts.

Tvrtko