[Piglit] [PATCH v2 3/5] util: Add a -list-subtests option that will list all the subtests

[Kenneth Graunke]

On 10/15/2013 05:32 PM, Ian Romanick wrote:
> From: Ian Romanick <ian.d.romanick at intel.com>
> 
> The utility is that all.tests can do 'sometest -list-subtests' to
> automatically get the list of subtests to run.  Since the syntax for
> selecting subtests is regularized, this works quite well.
> 
> Signed-off-by: Ian Romanick <ian.d.romanick at intel.com>
> Cc: Chad Versace <chad.versace at linux.intel.com>
> ---
>  tests/util/piglit-framework-gl.c | 15 +++++++++++++++
>  tests/util/piglit-framework-gl.h |  8 ++++++++
>  2 files changed, 23 insertions(+)

Sorry for coming late to the party on this one, but...a few comments:

I really like the idea of having consistent --subtest <foo> and
--list-subtests <foo> command line options.  It would be great to have a
consistent -h/--help as well (which would probably list subtests, when
available).

However, I am very concerned about the plan to use --list-subtests to
enumerate things in all.tests.

As I understand your current plan, piglit-run would have to invoke the
binary of every test that has subtests in order to get the list of
tests.  As we convert more tests to use subtests (or add new ones), this
becomes progressively more overhead: it has to run dozens of binaries
before it can even start running tests.

That said, processing all.tests already involves trawling all over disk
to find .frag, .vert, .geom, and .shader_test files, which is kind of
slow too.  Which is one reason some of us have been thinking about
moving the "produce the list of tests" step to be part of the build
process, instead of at every piglit-run startup.

But if the build process involved calling --list-subtests, then we would
need binaries built for both the host and the target systems, which
makes cross-compiling a nightmare.  I'd really like to avoid that.

So I think I would prefer to see us continue to explicitly list subtests.

--Ken