[Intel-gfx] [igt] Making the test-suite easier to run

[Mateo Lozano, Oscar]

> -----Original Message-----
> From: intel-gfx-bounces at lists.freedesktop.org [mailto:intel-gfx-
> bounces at lists.freedesktop.org] On Behalf Of Daniel Vetter
> Sent: Friday, November 15, 2013 5:23 PM
> To: Lespiau, Damien
> Cc: intel-gfx at lists.freedesktop.org
> Subject: Re: [Intel-gfx] [igt] Making the test-suite easier to run
> 
> On Fri, Nov 15, 2013 at 04:33:17PM +0000, Damien Lespiau wrote:
> > The objective of this series is to make the test-suite easier to run
> > by embedding a copy a piglit and providing porcelain on top of it in
> > the form of a makefile target. Beside python, there's no external
> > dependency to run the test suite after this series.
> >
> > The provided makefile target runs the full test suite. I'm still
> > interested in providing, as a follow-up of this work, shorcuts for
> > subsets of the testsuite that can be useful for developpers, subset
> > what we would maintain in tree. For
> > instance:
> >   - tests selected by topic: running gem_.* Vs kms_.* Vs pm_.* (or maybe
> when
> >     running, say, gt tests, exlude ^kms_.* ^pm_.* so we still still a lot of
> >     the other tests (core_, drm_, ...)
> >   - quick subtests (where we disable long running stress, race, ...
> > tests)
> >
> > patches 21-23 are something a bit different, try to pave the way for
> > quick runs (by really the first tiny step).
> 
> I'm ok with doing the slow/quick thing if people like it.o
> 
> On the real deal (i.e. patches 1-20) a few overall comments:
> 
> - I think we need to make it very clear that piglit should still be
>   developed upstream. So local changes shouldn't be allowed at all imo.
>   Afaics that would only affect tests/igt.tests - can't we fix that by
>   replicating the symlink, too?
> 
> - The makefile target looks like a script. I think it'd be better to
>   extract it as a real script.

+1 on that. The make target is not a valid option for Android (If I manage to run python on Android, I mean...)

> - I'm not too terribly sold on the convenience script. Imo it shouldn't be
>   more than executable documentation, since I'm a bit afraid that we'll
>   add neat features (like fancy resume with auto-blocking of crashing
>   tests) to it that would better be done in upstream piglit to benefit
>   everyone.
> 
> But overall I agree that we need a local copy of piglit since apparently way too
> few people on our team us it to run igts. And running igts without piglit is just
> nuts.
> 
> Cheers, Daniel
> >
> > The README has been updated and I copy/paste it here the
> documentation
> > for what would be the new way to run tests:
> >
> >         After having compiled the tests, one can run the test-suite with:
> >
> >         $ sudo make run-tests
> >
> >         "make run-tests" create a $date-piglit-results.$n directory with the
> >         results of the run. More specifically:
> >           - $date-piglit-results.$n/main JSON file with the test results
> >           - $date-piglit-results.$n/html/index.html HTML summary of
> > the run
> >
> >         Where $date is the date formated with `date +%Y%m%d` and $n the
> nth run
> >         of the day.
> >
> >         PIGLIT_FLAGS can be used to give options to the underlying piglit
> >         runner. For instance, to exclude test matching '^kms_':
> >
> >         $ sudo make run-tests PIGLIT_FLAGS="-x ^kms_"
> >
> >         For the list of piglit options, run:
> >
> >         $ ./piglit/piglit-run.py -h
> >
> >         Another useful feature is to be able to resume an interrupted run. To
> >         do that, make run-tests needs to know which run we are talking about:
> >
> >         $ sudo make run-tests RESUME=$date-piglit-results.$n
> >
> >         or, more succinctly:
> >
> >         $ sudo make run-tests R=$date-piglit-results.$n
> >
> >         It's possible to combine PIGLIT_FLAGS and RESUME. This is useful to
> >         resume runs where a specific test deterministically hang the machine:
> >
> >         $ sudo make run-tests PIGLIT_FLAGS="-x drv_module_reload"
> > R=$date-piglit-results.$n
> >
> > HTH,
> >
> > --
> > Damien
> >
> > _______________________________________________
> > Intel-gfx mailing list
> > Intel-gfx at lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/intel-gfx
> 
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> +41 (0) 79 365 57 48 - http://blog.ffwll.ch
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/intel-gfx

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47