[igt-dev] [PATCH i-g-t 0/6] Dynamic subtests

[Petri Latvala]

On Thu, Jun 20, 2019 at 08:47:12AM +0100, Tvrtko Ursulin wrote:
> 
> On 19/06/2019 12:51, Petri Latvala wrote:
> > Have you ever had a massive list of named things that you'd quite like
> > to test with separate subtests but maintaining a compile-time
> > exhaustive list of those things seems impossible, stupid, or both?
> > 
> > Have you ever wished you could just go over a list of things at
> > runtime and create subtests on the fly?
> > 
> > Have you ever looked at a long list of SKIP-results and wondered if it
> > makes sense to keep plopping those in CI results?
> > 
> > If so, this is your lucky day, for dynamic subtests are here!
> > 
> > While the restriction remains that "execution entry points" must still
> > be statically available no matter what the runtime environment and
> > configuration is, dynamic subtest containers are special subtest-like
> > entry points that can contain dynamic subtests. For example:
> > 
> > With normal subtests:
> > for_each_pipe_static(pipe)
> >    igt_subtest_f("fudge-with-pipe-%s", kmstest_pipe_name(pipe)) {
> >      int fd = drm_open_driver(DRIVER_ANY);
> >      igt_require(actually_have_this_pipe(fd, pipe));
> >      igt_assert(do_stuff(fd, pipe));
> >    }
> > 
> > 
> > With dynamic subtests:
> > igt_subtest_container("fudge")
> >    int fd = drm_open_driver(DRIVER_ANY);
> >    for_each_pipe(fd, pipe) {
> >      igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) {
> >        igt_assert(do_stuff(fd, pipe));
> >      }
> >    }
> > 
> 
> How would the below be converted:
> 
> int fd = drm_open_driver(DRIVER_ANY);
> 
> for_each_pipe_static(pipe) {
>   igt_subtest_group {
>     igt_fixture {
>       igt_require(actually_have_this_pipe(fd, pipe));
>     }	
> 
>     igt_subtest_f("fudge-with-pipe-%s", pipe_name(pipe))
>       fudge(fd, pipe);
> 
>     igt_subtest_f("kludge-with-pipe-%s", pipe_name(pipe))
>       kludge(fd, pipe);
>   }
> }
> 


igt_dynamic_subtest_container("fudge")
  for_each_pipe(fd, pipe)
    igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe))
      fudge(fd, pipe);

igt_dynamic_subtest_container("kludge")
  for_each_pipe(fd, pipe)
    igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe))
      kludge(fd, pipe);



Then you get results for "subtests" called igt at binaryname@fudge and
igt at binaryname@fudge at pipe-a and igt at binaryname@fudge at pipe-b, along
with igt at binaryname@kludge and igt at binaryname@kludge at pipe-a and
igt at binaryname@kludge at pipe-b



> ?
> 
> > 
> > Running on hardware that only has pipes A-C? pipe-d doesn't even get
> > attempted.
> > 
> > 
> > Semantics
> > =========
> > 
> > Dynamic subtests can only appear in a subtest container. The name is
> > up to debate: This series calls it igt_subtest_container().
> 
> igt_dynamic_subtest_group?
> 
> > A dynamic subtest container can only contain dynamic subtests, it
> > cannot do failures on its own. In other words:
> >   - igt_assert not allowed
> >   - igt_skip / igt_require is allowed though
> 
> Can it do fixtures?

No.

For shared initialization, do

igt_dynamic_subtest_container("hello-init") {
  /* right here */
  int fd = drm_open_driver(DRIVER_ANY);
  init_stuffs(fd);
  /* init done */

  for (......) {
    igt_dynamic_subtest(...)
}


> 
> > Any failing dynamic subtest will result in the container reporting a
> > failure.
> > 
> > Not executing any dynamic subtests from a container will result in the
> > container reporting a SKIP automatically. Best practices: If possible,
> > instead of using igt_require in an igt_dynamic_subtest, just don't
> > enter it for such an occasion. In other words:
> > 
> > Do not:
> > for_each_pipe(fd, pipe)
> >    igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
> >      igt_require(pipe_has_things(pipe));
> >      ...
> >    }
> > 
> > 
> > Instead do:
> > for_each_pipe(fd, pipe) {
> >    if (!pipe_has_things(pipe))
> >      continue;
> >    igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
> >      ...
> >    }
> > }
> > 
> > That way, tests that currently carefully track the number of, say,
> > connected displays of type $x, to properly skip when their amount is
> > 0, will get their SKIP automatically instead.
> > 
> > Dynamic subtests are filterable: Just like --run-subtest, there's
> > --dynamic-subtest that takes glob expressions. This is for debugging
> > only, CI will not use them.
> > 
> > Dynamic subtests are _NOT_ listable. While it might be useful,
> > implementing listing requires another layer of igt_fixture usage
> > inside dynamic subtest containers and I'd rather have the code be
> > simple than have listing. The default of running all dynamic subtests
> > should make sense for most cases, and special cases (like debugging)
> > should be able to know already what they want to run.
> > 
> > 
> > Results in CI: CI will show results for both the container, and the
> > dynamic subtests within it. The naming is:
> > 
> > igt at binary@subcontainer  -  the container, has as its output the whole
> > shebang that dynamic subtests printed.
> > 
> > igt at binary@subcontainer at dynamicname  -  a dynamic subtest appears as a
> > separate name, WITHOUT grouping or nesting. Any relation to the
> > container will not be linked at this point. Possibly in the future
> > when a usability wizard figures out the best way to browse results...
> 
> Did not get this part. Is it the first or second option? Or both? If it is
> the second then CI has to handle dynamic subtests so why we even need the
> complication to start with?


It's both. The container will report a result (based on what the
dynamic tests did) and there's also a result for each dynamic subtest.

The complication is needed because with dynamic subtests the execution
entry and results reporting are two different sets, reported subtests
being a superset of the entries.

CI will handle (knocking on wood here) dynamic results fine in its
current state, because dynamic subtest results walk like a duck, smell
like a duck, and look like normal subtest results.


-- 
Petri Latvala