[Intel-gfx] [RFC i-g-t v2] igt: Test tagging support

[Daniel Vetter]

On Fri, Jul 21, 2017 at 2:59 PM, Chris Wilson <chris at chris-wilson.co.uk> wrote:
> My dream would be for the system to recognise which tests provide
> coverage for a patch and rank them by relevance and then run until death
> or timeout. Without such introspection, the best we can do is to have
> the developer supply the selection via a common system of tags, one
> approximation would be to supply tags based on altered files and
> functions. Hmm, that seems doable.

In principle I agree, in practice I haven't seen such an oracle that
actually works. There's not just the coverage problem, but there's a
huge process problem: If our oracle isn't perfect at constructing the
test set, or if it's timeout based, there's a good chance that even
the same (or very similar patch) will run different sets of testcases.
That adds noise to CI results, and noise trains developers to ignore
it (and no amount of reminding to look at detailed results will fix
that).

The other problem is that we need to be able to compare those test
runs against a baseline. That means we must run the full regression
test set on drm-tip, or pre-merge can't give you results. And drm-tip
updates a few times a day already. We could probably batch more, but
we must have enough machine time to get through a few full runs,
without even taking pre-merge testing into account.

That means even the full set can't really take more than perhaps 6-8h,
even when you have a perfect oracle that selects the test set for each
patch.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch