[Intel-gfx] [RFC i-g-t] GEM features into feat_profile.json

[Daniel Vetter]

On Wed, May 31, 2017 at 03:02:18PM +0100, Chris Wilson wrote:
> On Wed, May 31, 2017 at 04:45:16PM +0300, Joonas Lahtinen wrote:
> > On ke, 2017-05-31 at 13:58 +0100, Chris Wilson wrote:
> > > On Wed, May 31, 2017 at 03:23:12PM +0300, Joonas Lahtinen wrote:
> > > > 
> > > > Hello,
> > > > 
> > > > I went through the gem_* tests from intel-gpu-tools and categorized
> > > > them into roughly categories "X | X robustness | X performance" ready
> > > > to be added to the feat_profile.json.
> > > > 
> > > > Lets open a discussion which ones should go where. I tried to place a
> > > > single test to under only one category and I'm kind of hopeful that
> > > > we'll have the ability to add "depends_on" to create super features in
> > > > the future, instead of placing a single test under multiple categories.
> > > > 
> > > > I didn't check all the subtests nor wildcard matching with other tests,
> > > > this is just all the test names placed under some categories.
> > > 
> > > You seem to have assigned them exclusively to one category or another,
> > > most tests belong to a few of these categories. More when you consider a
> > > subtest may be targetting a completely different aspect.
> > 
> > Yes, that's what I meant to say :) Subtests should probably be matched
> > by another pattern like "\btiled\b", "\bflink\b" etc.
> > 
> > Ultimately there would be a resolver which would re-assign the
> > subtests:
> > 
> > "Global objects" would then get:
> > 
> > 	"include_subtests": "flink",
> > 
> > Which would steal subtests with /\bflink\b/ from tests. Do we agree
> > that one subtest would be assigned to one category only, or do you
> > want to see duplication even at that level?
> 
> I see duplication everywhere. It's more a concept of tags as opposed to
> categories.
> 
> The use of such a system would be as
> 	"give me all the tests that exercise relocation"
> 	"give me all the tests that use a context"
> 	"give me all the tests that exercise contention on $mutex"
> 	"give me all the tests that exercise file.c:line / this patch set"

file/line is probably out of scope for this, for that we'd need to have
coverage information for each testcase, and then we could map a diff to
the relevant set of testcases. Nifty, but probably an awful lot of work to
automate ...

> The last one especially.

Yeah that was kinda the idea of this, the entries wouldn't be only an
exclusive list of tests only (piglit does that grouping already), but
match patterns for a given feature. And a given test can easily show up in
multiple features (e.g. a testcase that tests gpu hangs vs. suspend/resume
obviously needs to be both a hangcheck test and an s/r test).

I think we can just extend stuff though.

Anyway, thanks a lot for kickstarting this, unfortunately my vacations
start in like 4h so can't really digg into this more before heading back.

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch