[Intel-gfx] [PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune the list of tests to run

[Daniel Vetter]

On Tue, Mar 26, 2013 at 10:00:23AM -0700, Ben Widawsky wrote:
> On Tue, Mar 26, 2013 at 04:38:58PM +0100, Daniel Vetter wrote:
> > On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> > > There are two ways to run tests, directly with make check/test
> > > or through piglit.
> > > 
> > > When IGT_SIMULATION is set to '1', we substitute the list of tests in
> > > those two code paths with carefully selected tests. The stress tests and
> > > other horrors are left to torture the real hardware as they don't make
> > > too much sense in simulation.
> > > 
> > > Signed-off-by: Damien Lespiau <damien.lespiau at intel.com>
> > 
> > This feels a bit fragile since even just now I often fail to put a test
> > newly converted to the subtest stuff into the right single/multi make
> > target list. Also, this way we exclude a test by default, which also feels
> > like the wrong way round (most of the tests we're adding aren't heavy
> > stress-tests, but more excercise some corner-case).
> > 
> > So what about a sprinkling
> > 
> > drmtest_skip_on_simulation();
> > 
> > over the remaining testcases instead, which just calls exit(77);? Only
> > caveat is that for subtests we need to put it into the right spot to not
> > break subtest enumeration in piglit (it's a bit fragile).
> > 
> > Cheers, Daniel
> 
> I prefer Damien's explicit definitions in a file. It makes it much
> easier to see exactly what's run, and add a test as needed. Since
> simulation testing will always be a massive subset of the whole i-g-t
> suite, and we'll probably only rarely add or remove a test, I think it's
> not a big concern that we might miss a test.

Imo that's the wrong approach, since most of the tests we've recently
added exercise corner-cases of our code, and in a rather deterministic
way. If we currently have too many tests to get through all of the useful
ones in a day, we need more machines, not fewer tests.

In the end we want to be able to run full piglit on all this stuff in
simulation after all. Cc'ing Yi so he knows where I'm aiming at ;-)
-Daniel

> 
> > 
> > > ---
> > >  tests/Makefile.am | 43 ++++++++++++++++++++++++++++++++++++++++---
> > >  1 file changed, 40 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/tests/Makefile.am b/tests/Makefile.am
> > > index 2fddfe8..33db962 100644
> > > --- a/tests/Makefile.am
> > > +++ b/tests/Makefile.am
> > > @@ -137,22 +137,59 @@ kernel_tests = \
> > >  	$(multi_kernel_tests) \
> > >  	$(NULL)
> > >  
> > > +# We maintain here a list of tests to run in our simulation environments
> > > +single_simulation_tests = \
> > > +	gem_mmap \
> > > +	gem_pread_after_blit \
> > > +	gem_ring_sync_loop \
> > > +	gem_ctx_basic \
> > > +	gem_pipe_control_store_loop \
> > > +	gem_storedw_loop_render \
> > > +	gem_storedw_loop_blt \
> > > +	gem_storedw_loop_bsd \
> > > +	gem_render_linear_blits \
> > > +	gem_tiled_blits \
> > > +	gem_cpu_reloc \
> > > +	$(NULL)
> > > +
> > > +multi_simulation_tests = \
> > > +	gem_exec_nop \
> > > +	gem_mmap_gtt \
> > > +	$(NULL)
> > > +
> > > +simulation_tests = \
> > > +	$(single_simulation_tests) \
> > > +	$(multi_simulation_tests) \
> > > +	$(NULL)
> > > +
> > >  TESTS = \
> > >  	$(NULL)
> > >  
> > >  test:
> > >  	@whoami | grep root || ( echo ERROR: not running as root; exit 1 )
> > >  	@./check_drm_clients
> > > -	@make TESTS="${kernel_tests}" check
> > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > +		make TESTS="${simulation_tests}" check; \
> > > +	else \
> > > +		make TESTS="${kernel_tests}" check; \
> > > +	fi
> > >  
> > >  list-single-tests:
> > >  	@echo TESTLIST
> > > -	@echo ${single_kernel_tests}
> > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > +		echo ${single_simulation_tests}; \
> > > +	else \
> > > +		echo ${single_kernel_tests}; \
> > > +	fi
> > >  	@echo END TESTLIST
> > >  
> > >  list-multi-tests:
> > >  	@echo TESTLIST
> > > -	@echo ${multi_kernel_tests}
> > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > +		echo ${multi_simulation_tests}; \
> > > +	else \
> > > +		echo ${multi_kernel_tests}; \
> > > +	fi
> > >  	@echo END TESTLIST
> > >  
> > >  HANG = \
> > > -- 
> > > 1.7.11.7
> > > 
> > > _______________________________________________
> > > Intel-gfx mailing list
> > > Intel-gfx at lists.freedesktop.org
> > > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> > 
> > -- 
> > Daniel Vetter
> > Software Engineer, Intel Corporation
> > +41 (0) 79 365 57 48 - http://blog.ffwll.ch
> > _______________________________________________
> > Intel-gfx mailing list
> > Intel-gfx at lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> 
> -- 
> Ben Widawsky, Intel Open Source Technology Center

-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch