[igt-dev] [PATCH i-g-t 0/6] Dynamic subtests
Petri Latvala
petri.latvala at intel.com
Wed Jun 19 11:51:01 UTC 2019
Have you ever had a massive list of named things that you'd quite like
to test with separate subtests but maintaining a compile-time
exhaustive list of those things seems impossible, stupid, or both?
Have you ever wished you could just go over a list of things at
runtime and create subtests on the fly?
Have you ever looked at a long list of SKIP-results and wondered if it
makes sense to keep plopping those in CI results?
If so, this is your lucky day, for dynamic subtests are here!
While the restriction remains that "execution entry points" must still
be statically available no matter what the runtime environment and
configuration is, dynamic subtest containers are special subtest-like
entry points that can contain dynamic subtests. For example:
With normal subtests:
for_each_pipe_static(pipe)
igt_subtest_f("fudge-with-pipe-%s", kmstest_pipe_name(pipe)) {
int fd = drm_open_driver(DRIVER_ANY);
igt_require(actually_have_this_pipe(fd, pipe));
igt_assert(do_stuff(fd, pipe));
}
With dynamic subtests:
igt_subtest_container("fudge")
int fd = drm_open_driver(DRIVER_ANY);
for_each_pipe(fd, pipe) {
igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) {
igt_assert(do_stuff(fd, pipe));
}
}
Running on hardware that only has pipes A-C? pipe-d doesn't even get
attempted.
Semantics
=========
Dynamic subtests can only appear in a subtest container. The name is
up to debate: This series calls it igt_subtest_container().
A dynamic subtest container can only contain dynamic subtests, it
cannot do failures on its own. In other words:
- igt_assert not allowed
- igt_skip / igt_require is allowed though
Any failing dynamic subtest will result in the container reporting a
failure.
Not executing any dynamic subtests from a container will result in the
container reporting a SKIP automatically. Best practices: If possible,
instead of using igt_require in an igt_dynamic_subtest, just don't
enter it for such an occasion. In other words:
Do not:
for_each_pipe(fd, pipe)
igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
igt_require(pipe_has_things(pipe));
...
}
Instead do:
for_each_pipe(fd, pipe) {
if (!pipe_has_things(pipe))
continue;
igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
...
}
}
That way, tests that currently carefully track the number of, say,
connected displays of type $x, to properly skip when their amount is
0, will get their SKIP automatically instead.
Dynamic subtests are filterable: Just like --run-subtest, there's
--dynamic-subtest that takes glob expressions. This is for debugging
only, CI will not use them.
Dynamic subtests are _NOT_ listable. While it might be useful,
implementing listing requires another layer of igt_fixture usage
inside dynamic subtest containers and I'd rather have the code be
simple than have listing. The default of running all dynamic subtests
should make sense for most cases, and special cases (like debugging)
should be able to know already what they want to run.
Results in CI: CI will show results for both the container, and the
dynamic subtests within it. The naming is:
igt at binary@subcontainer - the container, has as its output the whole
shebang that dynamic subtests printed.
igt at binary@subcontainer at dynamicname - a dynamic subtest appears as a
separate name, WITHOUT grouping or nesting. Any relation to the
container will not be linked at this point. Possibly in the future
when a usability wizard figures out the best way to browse results...
Existing tests that are good candidates for conversion to dynamic
subtests:
1) Anything using for_each_pipe_static
2) Tests that have one subtest per i915 engine
3) Kernel selftests
Petri Latvala (6):
lib: Introduce dynamic subtests
lib/tests: Unit tests for dynamic subtests
runner/resultgen: Refactor output parsing
runner/json_tests: Adapt to better output parsing
runner: Parse dynamic subtest outputs and results
runner/json_tests: Test dynamic subtests
lib/igt_core.c | 119 +++-
lib/igt_core.h | 89 +++
lib/tests/igt_dynamic_subtests.c | 186 ++++++
lib/tests/meson.build | 1 +
runner/job_list.c | 11 +
runner/job_list.h | 3 +
.../aborted-after-a-test/reference.json | 2 +-
.../dmesg-escapes/reference.json | 2 +-
.../dmesg-results/reference.json | 8 +-
.../reference.json | 4 +-
.../reference.json | 4 +-
.../dmesg-warn-level/reference.json | 4 +-
.../dynamic-subtests/0/dmesg.txt | 7 +
.../dynamic-subtests/0/err.txt | 36 ++
.../dynamic-subtests/0/journal.txt | 2 +
.../dynamic-subtests/0/out.txt | 19 +
.../dynamic-subtests/1/dmesg.txt | 5 +
.../dynamic-subtests/1/err.txt | 2 +
.../dynamic-subtests/1/journal.txt | 2 +
.../dynamic-subtests/1/out.txt | 5 +
.../dynamic-subtests/2/dmesg.txt | 10 +
.../dynamic-subtests/2/err.txt | 6 +
.../dynamic-subtests/2/journal.txt | 4 +
.../dynamic-subtests/2/out.txt | 8 +
.../dynamic-subtests/README.txt | 2 +
.../dynamic-subtests/endtime.txt | 1 +
.../dynamic-subtests/joblist.txt | 3 +
.../dynamic-subtests/metadata.txt | 12 +
.../dynamic-subtests/reference.json | 156 +++++
.../dynamic-subtests/starttime.txt | 1 +
.../dynamic-subtests/uname.txt | 1 +
.../json_tests_data/normal-run/reference.json | 8 +-
.../reference.json | 2 +-
.../notrun-results/reference.json | 2 +-
.../piglit-style-dmesg/reference.json | 8 +-
.../warnings-with-dmesg-warns/reference.json | 8 +-
.../json_tests_data/warnings/reference.json | 8 +-
runner/output_strings.h | 29 +-
runner/resultgen.c | 589 ++++++++++++------
runner/runner_json_tests.c | 3 +-
40 files changed, 1148 insertions(+), 224 deletions(-)
create mode 100644 lib/tests/igt_dynamic_subtests.c
create mode 100644 runner/json_tests_data/dynamic-subtests/0/dmesg.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/0/err.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/0/journal.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/0/out.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/1/dmesg.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/1/err.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/1/journal.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/1/out.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/2/dmesg.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/2/err.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/2/journal.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/2/out.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/README.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/endtime.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/joblist.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/metadata.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/reference.json
create mode 100644 runner/json_tests_data/dynamic-subtests/starttime.txt
create mode 100644 runner/json_tests_data/dynamic-subtests/uname.txt
--
2.19.1
More information about the igt-dev
mailing list