[Intel-gfx] [RFC i-g-t v2] igt: Test tagging support

[Jani Nikula]

On Fri, 21 Jul 2017, Tvrtko Ursulin <tursulin at ursulin.net> wrote:
> Proposal to add test tags as a replacement for separate test
> list which can be difficult to maintain and get out of date.
>
> Putting this maintanenace inline with tests makes it easier
> to remember to update the (now implicit) lists, and also enables
> richer test selection possibilities for the test runner.
>
> Current method of implying tags from test/subtest names has a
> couple of problems one of which is where some names can use
> the same token for different meanings. (One example is the
> "default" token.) It also creates a name clash between naming
> and tagging.

I'll say this once more, and then I'll shut up.

I think it's fundamentally the wrong thing to do to put the tags in the
test sources themselves. It should be enough to look at the commit churn
we had to move tests to/from BAT before we had test lists to be
convinced. You shouldn't have to modify the tests to tag/untag them.

You'll also find it'll be an interesting and needed feature to be able
to have tags that aren't committed to the public repo. This would fail
completely if you needed to have commits touching all the tests you need
to tag. And we don't want to maintain both the test list and the tagging
thing.

This is the "I told you so" mail I'll reference in the future.


BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center