[Intel-gfx] [RFC i-g-t v2] igt: Test tagging support

[Chris Wilson]

Quoting Martin Peres (2017-07-21 13:02:47)
> On 21/07/17 14:27, Chris Wilson wrote:
> > Quoting Tvrtko Ursulin (2017-07-21 12:08:00)
> >>
> >> On 21/07/2017 11:36, Chris Wilson wrote:
> >>> Quoting Tvrtko Ursulin (2017-07-21 11:20:05)
> >>>> From: Tvrtko Ursulin <tvrtko.ursulin at intel.com>
> >>
> >> [snip]
> >>
> >>>> --- a/tests/gem_concurrent_all.c
> >>>> +++ b/tests/gem_concurrent_all.c
> >>>> @@ -1492,47 +1492,47 @@ run_mode(const char *prefix,
> >>>>                           igt_subtest_group  {
> >>>>                                   igt_fixture p->require();
> >>>>    
> >>>> -                               igt_subtest_f("%s-%s-%s-sanitycheck0%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
> >>>> +                               igt_gem_stress_subtest_f("", "%s-%s-%s-sanitycheck0%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
> >>>
> >>> They are not all stress tests. So you want to be able to build the tags
> >>> dynamically... Similarly they offer different types of "stress", you
> >>> probably don't want to lump the hang tests in amongst thes plain
> >>> concurrency tests, and you probably want the swapping tests separated
> >>> etc. Stress is missing the point.
> >>
> >> Dynamic tags are doable. If you just wanted to include "stress"
> >> dynamically current RFC can already do that.
> >>
> >>          igt_gem_subtest_f(is_stress ? "stress" : "", name, ...)
> >>
> >> If you wanted a dynamic set of multiple tags that could be added as well
> >> I guess. Like a flag based control of "stress", "swapping", "hang",
> >> "basic", or something. How nice or ugly API depends on the actual
> >> requirements.
> > 
> > hang, swap, shrink, gtt, wc, cpu, pwrite, pread, contexts, fds, prime,
> > dmabuf and many more when you start looking for the complete set of
> > tags/keywords/categories.
> 
> This is why I would rather use the execution time of tests as a way to 
> tag the tests. What I want is to have a couple of options that brings me 
> the best coverage in a certain amount of time:
>   - BAT: 70% coverage in 10 minutes
>   - FULL: 95% coverage in 6 hours
>   - Stress: 99% coverage in 1 year
> 
> What do the tags hang, swap, shrink, gtt, etc.. are supposed to bring to me?

Nothing. CI should be focused purely on bang per buck. Until we can
measure coverage in a meaningful way, evaluating tests is entirely
arbitrary and worse we have no rigorous means for developing tests.

My dream would be for the system to recognise which tests provide
coverage for a patch and rank them by relevance and then run until death
or timeout. Without such introspection, the best we can do is to have
the developer supply the selection via a common system of tags, one
approximation would be to supply tags based on altered files and
functions. Hmm, that seems doable.
-Chris