[Intel-gfx] [RFC i-g-t 0/4] Redundant test pruning

[Daniel Vetter]

On Tue, Jun 27, 2017 at 09:02:02AM +0100, Tvrtko Ursulin wrote:
> 
> On 26/06/2017 17:09, Daniel Vetter wrote:
> > On Fri, Jun 23, 2017 at 12:31:39PM +0100, Tvrtko Ursulin wrote:
> > > From: Tvrtko Ursulin <tvrtko.ursulin at intel.com>
> > > 
> > > Small series which saves test execution time by removing the redundant tests.
> > > 
> > > Tvrtko Ursulin (4):
> > >    igt: Remove default from the engine list
> > >    gem_exec_basic: Exercise the default engine selection
> > >    gem_sync: Add all and store_all subtests
> > >    extended.testlist: Remove some test-subtest combinations
> > 
> > Ack on patches 1&2, but I'm not sold on patch 3. Atm gem_* takes a
> > ridiculous amount of machine time to run, you're adding more stuff. Are
> > those tests really drastially better at catching races if we run them 10x
> > longer? Is there no better way to exercise the races (lots more machines,
> > maybe slower ones, which is atm impossible since it just takes way, way
> > too long and we need an entire farm just for one machine).
> 
> New gem_sync subtests were suggested by Chris after I send the first version
> of the series with the goal of getting the same coverage in faster time.
> 
> If you look at patch 4, it removes 18 * 150s of gem_sync subtests, and adds
> 4 * 150s. So in total we are 35 minutes better of in the best case, a bit
> less on smaller machines.

So why keep the other 18 tests when we have coverage by the new ones? Some
developer modes (like e.g. kms_frontbuffer_tracking has) for testing is
all nice, but piling ever higher amounts of redundant tests up isn't great
imo.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch