[Intel-gfx] [PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune the list of tests to run

[Sun, Yi]

Hi Daniel, 

Thank you for your notification. This would make it easy to run sub suite on simulation during this stage. I know the goal is to run the whole i-g-t test case. 
So we can run tests/igt.test just like any other real machines. Ok, I'll add it to our nightly ASAP.

Thanks
    --Sun, Yi


> -----Original Message-----
> From: Daniel Vetter [mailto:daniel.vetter at ffwll.ch] On Behalf Of Daniel Vetter
> Sent: Wednesday, March 27, 2013 8:25 AM
> To: Ben Widawsky
> Cc: Daniel Vetter; Lespiau, Damien; intel-gfx at lists.freedesktop.org; Sun, Yi
> Subject: Re: [Intel-gfx] [PATCH i-g-t 3/5] tests: Use IGT_SIMULATION to tune
> the list of tests to run
> 
> On Tue, Mar 26, 2013 at 10:00:23AM -0700, Ben Widawsky wrote:
> > On Tue, Mar 26, 2013 at 04:38:58PM +0100, Daniel Vetter wrote:
> > > On Tue, Mar 26, 2013 at 03:24:37PM +0000, Damien Lespiau wrote:
> > > > There are two ways to run tests, directly with make check/test or
> > > > through piglit.
> > > >
> > > > When IGT_SIMULATION is set to '1', we substitute the list of tests
> > > > in those two code paths with carefully selected tests. The stress
> > > > tests and other horrors are left to torture the real hardware as
> > > > they don't make too much sense in simulation.
> > > >
> > > > Signed-off-by: Damien Lespiau <damien.lespiau at intel.com>
> > >
> > > This feels a bit fragile since even just now I often fail to put a
> > > test newly converted to the subtest stuff into the right
> > > single/multi make target list. Also, this way we exclude a test by
> > > default, which also feels like the wrong way round (most of the
> > > tests we're adding aren't heavy stress-tests, but more excercise some
> corner-case).
> > >
> > > So what about a sprinkling
> > >
> > > drmtest_skip_on_simulation();
> > >
> > > over the remaining testcases instead, which just calls exit(77);?
> > > Only caveat is that for subtests we need to put it into the right
> > > spot to not break subtest enumeration in piglit (it's a bit fragile).
> > >
> > > Cheers, Daniel
> >
> > I prefer Damien's explicit definitions in a file. It makes it much
> > easier to see exactly what's run, and add a test as needed. Since
> > simulation testing will always be a massive subset of the whole i-g-t
> > suite, and we'll probably only rarely add or remove a test, I think
> > it's not a big concern that we might miss a test.
> 
> Imo that's the wrong approach, since most of the tests we've recently added
> exercise corner-cases of our code, and in a rather deterministic way. If we
> currently have too many tests to get through all of the useful ones in a day, we
> need more machines, not fewer tests.
> 
> In the end we want to be able to run full piglit on all this stuff in simulation after
> all. Cc'ing Yi so he knows where I'm aiming at ;-) -Daniel
> 
> >
> > >
> > > > ---
> > > >  tests/Makefile.am | 43
> > > > ++++++++++++++++++++++++++++++++++++++++---
> > > >  1 file changed, 40 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/tests/Makefile.am b/tests/Makefile.am index
> > > > 2fddfe8..33db962 100644
> > > > --- a/tests/Makefile.am
> > > > +++ b/tests/Makefile.am
> > > > @@ -137,22 +137,59 @@ kernel_tests = \
> > > >  	$(multi_kernel_tests) \
> > > >  	$(NULL)
> > > >
> > > > +# We maintain here a list of tests to run in our simulation
> > > > +environments single_simulation_tests = \
> > > > +	gem_mmap \
> > > > +	gem_pread_after_blit \
> > > > +	gem_ring_sync_loop \
> > > > +	gem_ctx_basic \
> > > > +	gem_pipe_control_store_loop \
> > > > +	gem_storedw_loop_render \
> > > > +	gem_storedw_loop_blt \
> > > > +	gem_storedw_loop_bsd \
> > > > +	gem_render_linear_blits \
> > > > +	gem_tiled_blits \
> > > > +	gem_cpu_reloc \
> > > > +	$(NULL)
> > > > +
> > > > +multi_simulation_tests = \
> > > > +	gem_exec_nop \
> > > > +	gem_mmap_gtt \
> > > > +	$(NULL)
> > > > +
> > > > +simulation_tests = \
> > > > +	$(single_simulation_tests) \
> > > > +	$(multi_simulation_tests) \
> > > > +	$(NULL)
> > > > +
> > > >  TESTS = \
> > > >  	$(NULL)
> > > >
> > > >  test:
> > > >  	@whoami | grep root || ( echo ERROR: not running as root; exit 1 )
> > > >  	@./check_drm_clients
> > > > -	@make TESTS="${kernel_tests}" check
> > > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > > +		make TESTS="${simulation_tests}" check; \
> > > > +	else \
> > > > +		make TESTS="${kernel_tests}" check; \
> > > > +	fi
> > > >
> > > >  list-single-tests:
> > > >  	@echo TESTLIST
> > > > -	@echo ${single_kernel_tests}
> > > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > > +		echo ${single_simulation_tests}; \
> > > > +	else \
> > > > +		echo ${single_kernel_tests}; \
> > > > +	fi
> > > >  	@echo END TESTLIST
> > > >
> > > >  list-multi-tests:
> > > >  	@echo TESTLIST
> > > > -	@echo ${multi_kernel_tests}
> > > > +	@if [ "${IGT_SIMULATION}" == 1 ]; then \
> > > > +		echo ${multi_simulation_tests}; \
> > > > +	else \
> > > > +		echo ${multi_kernel_tests}; \
> > > > +	fi
> > > >  	@echo END TESTLIST
> > > >
> > > >  HANG = \
> > > > --
> > > > 1.7.11.7
> > > >
> > > > _______________________________________________
> > > > Intel-gfx mailing list
> > > > Intel-gfx at lists.freedesktop.org
> > > > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> > >
> > > --
> > > Daniel Vetter
> > > Software Engineer, Intel Corporation
> > > +41 (0) 79 365 57 48 - http://blog.ffwll.ch
> > > _______________________________________________
> > > Intel-gfx mailing list
> > > Intel-gfx at lists.freedesktop.org
> > > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> >
> > --
> > Ben Widawsky, Intel Open Source Technology Center
> 
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> +41 (0) 79 365 57 48 - http://blog.ffwll.ch