[Piglit] [PATCH 3/4] util: Add a -list-subtests option that will list all the subtests

[Daniel Vetter]

On Fri, Oct 04, 2013 at 06:11:00PM -0700, Ian Romanick wrote:
> From: Ian Romanick <ian.d.romanick at intel.com>
> 
> This required some ugly hacking about to get the list of subtests to
> process_args.  Suggestions?
> 
> The utility is that all.tests can do 'sometest -list-subtests' to
> automatically get the list of subtests to run.  Since the syntax for
> selecting subtests is regularized, this works quite well.
> 
> Signed-off-by: Ian Romanick <ian.d.romanick at intel.com>
> Cc: Chad Versace <chad.versace at linux.intel.com>

Since intel-gpu-tools has a very similar --list-subtest and --run-subtest
I guess it can't hurt if I quickly explain what we have. Most parts are
very similar (e.g. the arg parsing stuff to setup the test suite), the big
difference is that we don't expect tests to supply a static list of
subtests. Instead there's rather magic

bool __igt_run_subtest(const char *subtest_name);

which returns true if the subtest should be run, false otherwise. In
normal mode that's all it does, but if the --list-subtest parameter has
been detected in the arg parsing function it will always return false but
instead print just print out all the testcase names. This is then wrapped
into some magic macros to create subtest blocks:

igt_subtest("foo") {
	/* code for subtest foo */
}

The clear downside is that the global code in tests needs to be written
more carefully (to avoid polluting the subtest list, among other things).
To help that we have more magic code blocks to mark such global setup
sections and prevent them from causing havoc when listing subtest:

igt_fixture {
	/* global setup code goes here, not run when enumerating subtests */
}

The really great upside is that it's trivially easy to construct subtests
programmatically at runtime. This is great if you have combinatorial tests
with twists, e.g.

for (fmt in formats) {
	for (flag = 0; flag < FLAG_MASK + 1; flag++) {
		igt_subtest_f("normal-%s%s%s%s", fmt_to_str(fmt),
			      flag & FLAG_FOO ? "-foo" : "",
			      flag & FLAG_BAR ? "-bar" : "",
			      flag & CRAZY ? "-crazy" : "")
			run_tests(fmt, flag);
		if (surface_sane(fmt) && !(flag & CRAZY)) {
			igt_subtest_f("normal-%s%s%s%s", fmt_to_str(fmt),
				      flag & FLAG_FOO ? "-foo" : "",
				      flag & FLAG_BAR ? "-bar" : "",
				      flag & CRAZY ? "-crazy" : "") {
				setup_different_env();
				run_tests(fmt, flag);
				teardown_different_env();
			}
		}
	} 
}

In igt we have ridiculous piles of tests structured like this where some
testscase with special flags can't be run in some special enviroments. So
the static list was something I've considered but then ditched again.

Anyway, just my 2 cents.

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch