[igt-dev] [PATCH i-g-t 0/6] Dynamic subtests
Tvrtko Ursulin
tvrtko.ursulin at linux.intel.com
Thu Jun 20 07:47:12 UTC 2019
On 19/06/2019 12:51, Petri Latvala wrote:
> Have you ever had a massive list of named things that you'd quite like
> to test with separate subtests but maintaining a compile-time
> exhaustive list of those things seems impossible, stupid, or both?
>
> Have you ever wished you could just go over a list of things at
> runtime and create subtests on the fly?
>
> Have you ever looked at a long list of SKIP-results and wondered if it
> makes sense to keep plopping those in CI results?
>
> If so, this is your lucky day, for dynamic subtests are here!
>
> While the restriction remains that "execution entry points" must still
> be statically available no matter what the runtime environment and
> configuration is, dynamic subtest containers are special subtest-like
> entry points that can contain dynamic subtests. For example:
>
> With normal subtests:
> for_each_pipe_static(pipe)
> igt_subtest_f("fudge-with-pipe-%s", kmstest_pipe_name(pipe)) {
> int fd = drm_open_driver(DRIVER_ANY);
> igt_require(actually_have_this_pipe(fd, pipe));
> igt_assert(do_stuff(fd, pipe));
> }
>
>
> With dynamic subtests:
> igt_subtest_container("fudge")
> int fd = drm_open_driver(DRIVER_ANY);
> for_each_pipe(fd, pipe) {
> igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) {
> igt_assert(do_stuff(fd, pipe));
> }
> }
>
How would the below be converted:
int fd = drm_open_driver(DRIVER_ANY);
for_each_pipe_static(pipe) {
igt_subtest_group {
igt_fixture {
igt_require(actually_have_this_pipe(fd, pipe));
}
igt_subtest_f("fudge-with-pipe-%s", pipe_name(pipe))
fudge(fd, pipe);
igt_subtest_f("kludge-with-pipe-%s", pipe_name(pipe))
kludge(fd, pipe);
}
}
?
>
> Running on hardware that only has pipes A-C? pipe-d doesn't even get
> attempted.
>
>
> Semantics
> =========
>
> Dynamic subtests can only appear in a subtest container. The name is
> up to debate: This series calls it igt_subtest_container().
igt_dynamic_subtest_group?
> A dynamic subtest container can only contain dynamic subtests, it
> cannot do failures on its own. In other words:
> - igt_assert not allowed
> - igt_skip / igt_require is allowed though
Can it do fixtures?
> Any failing dynamic subtest will result in the container reporting a
> failure.
>
> Not executing any dynamic subtests from a container will result in the
> container reporting a SKIP automatically. Best practices: If possible,
> instead of using igt_require in an igt_dynamic_subtest, just don't
> enter it for such an occasion. In other words:
>
> Do not:
> for_each_pipe(fd, pipe)
> igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
> igt_require(pipe_has_things(pipe));
> ...
> }
>
>
> Instead do:
> for_each_pipe(fd, pipe) {
> if (!pipe_has_things(pipe))
> continue;
> igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
> ...
> }
> }
>
> That way, tests that currently carefully track the number of, say,
> connected displays of type $x, to properly skip when their amount is
> 0, will get their SKIP automatically instead.
>
> Dynamic subtests are filterable: Just like --run-subtest, there's
> --dynamic-subtest that takes glob expressions. This is for debugging
> only, CI will not use them.
>
> Dynamic subtests are _NOT_ listable. While it might be useful,
> implementing listing requires another layer of igt_fixture usage
> inside dynamic subtest containers and I'd rather have the code be
> simple than have listing. The default of running all dynamic subtests
> should make sense for most cases, and special cases (like debugging)
> should be able to know already what they want to run.
>
>
> Results in CI: CI will show results for both the container, and the
> dynamic subtests within it. The naming is:
>
> igt at binary@subcontainer - the container, has as its output the whole
> shebang that dynamic subtests printed.
>
> igt at binary@subcontainer at dynamicname - a dynamic subtest appears as a
> separate name, WITHOUT grouping or nesting. Any relation to the
> container will not be linked at this point. Possibly in the future
> when a usability wizard figures out the best way to browse results...
Did not get this part. Is it the first or second option? Or both? If it
is the second then CI has to handle dynamic subtests so why we even need
the complication to start with?
Regards,
Tvrtko
>
>
> Existing tests that are good candidates for conversion to dynamic
> subtests:
>
>
> 1) Anything using for_each_pipe_static
>
> 2) Tests that have one subtest per i915 engine
>
> 3) Kernel selftests
>
>
>
>
> Petri Latvala (6):
> lib: Introduce dynamic subtests
> lib/tests: Unit tests for dynamic subtests
> runner/resultgen: Refactor output parsing
> runner/json_tests: Adapt to better output parsing
> runner: Parse dynamic subtest outputs and results
> runner/json_tests: Test dynamic subtests
>
> lib/igt_core.c | 119 +++-
> lib/igt_core.h | 89 +++
> lib/tests/igt_dynamic_subtests.c | 186 ++++++
> lib/tests/meson.build | 1 +
> runner/job_list.c | 11 +
> runner/job_list.h | 3 +
> .../aborted-after-a-test/reference.json | 2 +-
> .../dmesg-escapes/reference.json | 2 +-
> .../dmesg-results/reference.json | 8 +-
> .../reference.json | 4 +-
> .../reference.json | 4 +-
> .../dmesg-warn-level/reference.json | 4 +-
> .../dynamic-subtests/0/dmesg.txt | 7 +
> .../dynamic-subtests/0/err.txt | 36 ++
> .../dynamic-subtests/0/journal.txt | 2 +
> .../dynamic-subtests/0/out.txt | 19 +
> .../dynamic-subtests/1/dmesg.txt | 5 +
> .../dynamic-subtests/1/err.txt | 2 +
> .../dynamic-subtests/1/journal.txt | 2 +
> .../dynamic-subtests/1/out.txt | 5 +
> .../dynamic-subtests/2/dmesg.txt | 10 +
> .../dynamic-subtests/2/err.txt | 6 +
> .../dynamic-subtests/2/journal.txt | 4 +
> .../dynamic-subtests/2/out.txt | 8 +
> .../dynamic-subtests/README.txt | 2 +
> .../dynamic-subtests/endtime.txt | 1 +
> .../dynamic-subtests/joblist.txt | 3 +
> .../dynamic-subtests/metadata.txt | 12 +
> .../dynamic-subtests/reference.json | 156 +++++
> .../dynamic-subtests/starttime.txt | 1 +
> .../dynamic-subtests/uname.txt | 1 +
> .../json_tests_data/normal-run/reference.json | 8 +-
> .../reference.json | 2 +-
> .../notrun-results/reference.json | 2 +-
> .../piglit-style-dmesg/reference.json | 8 +-
> .../warnings-with-dmesg-warns/reference.json | 8 +-
> .../json_tests_data/warnings/reference.json | 8 +-
> runner/output_strings.h | 29 +-
> runner/resultgen.c | 589 ++++++++++++------
> runner/runner_json_tests.c | 3 +-
> 40 files changed, 1148 insertions(+), 224 deletions(-)
> create mode 100644 lib/tests/igt_dynamic_subtests.c
> create mode 100644 runner/json_tests_data/dynamic-subtests/0/dmesg.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/0/err.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/0/journal.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/0/out.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/1/dmesg.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/1/err.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/1/journal.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/1/out.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/2/dmesg.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/2/err.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/2/journal.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/2/out.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/README.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/endtime.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/joblist.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/metadata.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/reference.json
> create mode 100644 runner/json_tests_data/dynamic-subtests/starttime.txt
> create mode 100644 runner/json_tests_data/dynamic-subtests/uname.txt
>
More information about the igt-dev
mailing list