[Intel-gfx] [PATCH i-g-t v4 2/3] lib/igt_core: Unify handling of slow/combinatorial tests

[David Weinehall]

On Tue, Feb 16, 2016 at 04:45:26PM +0100, Daniel Vetter wrote:
> On Thu, Feb 11, 2016 at 01:09:33PM +0200, David Weinehall wrote:
> > Some subtests are not run by default, for various reasons;
> > be it because they're only for debugging, because they're slow,
> > or because they are not of high enough quality.
> > 
> > This patch aims to introduce a common mechanism for categorising
> > the subtests and introduces a flag (--all) that runs/lists all
> > subtests instead of just the default set.
> 
> Is the idea to also add a --test-flags interface? How is this meant to
> integrate with the current BAT runtime? Also we need to figure out
> how/whether we need to change the spec for how testrunners are supposed to
> run igt tests. Plus obviously patch igt.py in piglit.
> 
> I think I like where this is going, but I guess needs a bit more polish
> still. Integrating this into how we run/select BAT would be a great
> showcase of your concept I think.

The idea is to also add a --test-flags interface, yes, but I deemed it
outside the scope of this patchset.

The origin of this patch is a suggestion which I believe was
from you -- to handle combinatorial tests in one unified way instead of
using various different methods (--all, vs different test invocation
names).  Since the first patch got feedback that not all excluded
subtests were because they were slow I introduced the flags concept
(the patch still classifies everything non-default as slow though,
seeing as I'd rather leave such classification to the test authors).

Since neither BAT nor piglit at the moment uses gem_concurrent_blit[1] or
kms_frontbuffer_tracking --all, I don't think there's any need to patch
either of those at this point.

Until there's some agreement on a reasonable subset of
gem_concurrent_blit subtests that can be run within at most 30 seconds,
there's no possibility to have it included in BAT or piglit -- currently
even the default set of subtests (without passing --all) will take several
hours to complete.

There are a bunch of subtests that won't complete within 30 seconds even
when run on their own[2], which obviously complicates things.

FWIW, both Skylake and Broadwell hard-hangs when running
gem_concurrent_blit --all (and up until that point spews a mighty amount
of FAIL).


Kind regards, David

[1] gem_concurrent_blit is currently not run at all due to it being too
    slow. At the very least it would be nice to have a dedicated machine
    that ran it on a daily basis.  Having weekly runs with --all should
    be a longer-term goal, but with the current hangs and massive amount
    of FAILs that is not feasible yet.

[2] Notably the "swap-*" subtests.