[Intel-gfx] [RFC i-g-t 1/2] extended.testlist: Remove default and render engine test duplicates

[Daniel Vetter]

On Fri, Jun 23, 2017 at 12:21:48PM +0100, Chris Wilson wrote:
> Quoting Daniel Vetter (2017-06-23 12:14:49)
> > On Thu, Jun 22, 2017 at 12:45 PM, Tvrtko Ursulin
> > <tvrtko.ursulin at linux.intel.com> wrote:
> > >
> > > On 22/06/2017 10:54, Daniel Vetter wrote:
> > >>
> > >> On Fri, Jun 16, 2017 at 12:55 PM, Tvrtko Ursulin <tursulin at ursulin.net>
> > >> wrote:
> > >>>
> > >>> From: Tvrtko Ursulin <tvrtko.ursulin at intel.com>
> > >>>
> > >>> Where there is both default and render for the same test,
> > >>> remove the former to save some execution time.
> > >>
> > >>
> > >> If they are redundant, why do we even have them? Can we just remove
> > >
> > >
> > > Maybe we can remove the default entry from intel_execution_engines array?
> > > And just test that default == render explicitly in a few short tests.
> > 
> > That has my ack, in case you have the patch ...
> 
> No. The ABI does allow for the switch in theory and it is a distinct
> enum so it does need testing.

One testcase to make sure the default engine is the render engine is I
think ok. Testing all the combinatorics is over the top.

> > >> the testcase itself? Accumulating unused tests of questionable use at
> > >> best in igt is serious pain, because it means we never can get to a
> > >> world where new testcases are auto-added to CI withou some manual
> > >> review. And that's the world of pain we live in now and I really want
> > >> to get out of it. That means reviewing and removing testcases, not
> > >> massaging courated testlists forever, I don't think we have the time
> > >> for that among all the other tasks.
> > >>
> > >> </rant> :-)
> 
> It's the curated lists that are the problem, not the tests. The fact
> that we don't feel it is worthwhile to continuously test minor
> variations to catch weird bugs is our problem.

So it's an different topic, but I disagree. gem_* testcases are massively
over the top with how much machine time they blow through, to the point
where gem takes 90%+ of all machine time.

The result of that is that we simply don't run anything at all, so gem is
hurting kms_ and other test coverage. We really need to get this down to
something reasonable, and I think reasonable = completes within 1 work-day
on a fast machine. Stress-tests are ok to keep, and run on an ad-hoc
basis, but we can't have them in the default list.

Atm gem_concurrent_blt alone blows through 24 days of machine time, and
that's 24 days of not testing some other patch series. We have a too high
patch count to afford that, so if you insist on keeping all these gem
tests in the default set then we'll be able to merge about 1 patch series
per week. That just doesn't work.

Even excluding gem_concurrent_blt gem_* tests take a few times more than
everything else combined.

There's really imo just 2 options:
a) drastically reduce the time we spend running gem_* tests.
b) don't run gem_* tests, and don't merge gem_* patches.

Atm the result is that we can't even run the other igt tests, which means
everyone (not just the gem team) loses.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch