[igt-dev] [PATCH i-g-t 0/6] Dynamic subtests

[Tvrtko Ursulin]

On 20/06/2019 09:09, Petri Latvala wrote:
> On Thu, Jun 20, 2019 at 08:47:12AM +0100, Tvrtko Ursulin wrote:
>>
>> On 19/06/2019 12:51, Petri Latvala wrote:
>>> Have you ever had a massive list of named things that you'd quite like
>>> to test with separate subtests but maintaining a compile-time
>>> exhaustive list of those things seems impossible, stupid, or both?
>>>
>>> Have you ever wished you could just go over a list of things at
>>> runtime and create subtests on the fly?
>>>
>>> Have you ever looked at a long list of SKIP-results and wondered if it
>>> makes sense to keep plopping those in CI results?
>>>
>>> If so, this is your lucky day, for dynamic subtests are here!
>>>
>>> While the restriction remains that "execution entry points" must still
>>> be statically available no matter what the runtime environment and
>>> configuration is, dynamic subtest containers are special subtest-like
>>> entry points that can contain dynamic subtests. For example:
>>>
>>> With normal subtests:
>>> for_each_pipe_static(pipe)
>>>     igt_subtest_f("fudge-with-pipe-%s", kmstest_pipe_name(pipe)) {
>>>       int fd = drm_open_driver(DRIVER_ANY);
>>>       igt_require(actually_have_this_pipe(fd, pipe));
>>>       igt_assert(do_stuff(fd, pipe));
>>>     }
>>>
>>>
>>> With dynamic subtests:
>>> igt_subtest_container("fudge")
>>>     int fd = drm_open_driver(DRIVER_ANY);
>>>     for_each_pipe(fd, pipe) {
>>>       igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) {
>>>         igt_assert(do_stuff(fd, pipe));
>>>       }
>>>     }
>>>
>>
>> How would the below be converted:
>>
>> int fd = drm_open_driver(DRIVER_ANY);
>>
>> for_each_pipe_static(pipe) {
>>    igt_subtest_group {
>>      igt_fixture {
>>        igt_require(actually_have_this_pipe(fd, pipe));
>>      }	
>>
>>      igt_subtest_f("fudge-with-pipe-%s", pipe_name(pipe))
>>        fudge(fd, pipe);
>>
>>      igt_subtest_f("kludge-with-pipe-%s", pipe_name(pipe))
>>        kludge(fd, pipe);
>>    }
>> }
>>
> 
> 
> igt_dynamic_subtest_container("fudge")
>    for_each_pipe(fd, pipe)
>      igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe))
>        fudge(fd, pipe);
> 
> igt_dynamic_subtest_container("kludge")
>    for_each_pipe(fd, pipe)
>      igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe))
>        kludge(fd, pipe);
> 
> 
> 
> Then you get results for "subtests" called igt at binaryname@fudge and
> igt at binaryname@fudge at pipe-a and igt at binaryname@fudge at pipe-b, along
> with igt at binaryname@kludge and igt at binaryname@kludge at pipe-a and
> igt at binaryname@kludge at pipe-b
> 
> 
> 
>> ?
>>
>>>
>>> Running on hardware that only has pipes A-C? pipe-d doesn't even get
>>> attempted.
>>>
>>>
>>> Semantics
>>> =========
>>>
>>> Dynamic subtests can only appear in a subtest container. The name is
>>> up to debate: This series calls it igt_subtest_container().
>>
>> igt_dynamic_subtest_group?
>>
>>> A dynamic subtest container can only contain dynamic subtests, it
>>> cannot do failures on its own. In other words:
>>>    - igt_assert not allowed
>>>    - igt_skip / igt_require is allowed though
>>
>> Can it do fixtures?
> 
> No.
> 
> For shared initialization, do
> 
> igt_dynamic_subtest_container("hello-init") {
>    /* right here */
>    int fd = drm_open_driver(DRIVER_ANY);
>    init_stuffs(fd);
>    /* init done */
> 
>    for (......) {
>      igt_dynamic_subtest(...)
> }
> 
> 
>>
>>> Any failing dynamic subtest will result in the container reporting a
>>> failure.
>>>
>>> Not executing any dynamic subtests from a container will result in the
>>> container reporting a SKIP automatically. Best practices: If possible,
>>> instead of using igt_require in an igt_dynamic_subtest, just don't
>>> enter it for such an occasion. In other words:
>>>
>>> Do not:
>>> for_each_pipe(fd, pipe)
>>>     igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
>>>       igt_require(pipe_has_things(pipe));
>>>       ...
>>>     }
>>>
>>>
>>> Instead do:
>>> for_each_pipe(fd, pipe) {
>>>     if (!pipe_has_things(pipe))
>>>       continue;
>>>     igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
>>>       ...
>>>     }
>>> }
>>>
>>> That way, tests that currently carefully track the number of, say,
>>> connected displays of type $x, to properly skip when their amount is
>>> 0, will get their SKIP automatically instead.
>>>
>>> Dynamic subtests are filterable: Just like --run-subtest, there's
>>> --dynamic-subtest that takes glob expressions. This is for debugging
>>> only, CI will not use them.
>>>
>>> Dynamic subtests are _NOT_ listable. While it might be useful,
>>> implementing listing requires another layer of igt_fixture usage
>>> inside dynamic subtest containers and I'd rather have the code be
>>> simple than have listing. The default of running all dynamic subtests
>>> should make sense for most cases, and special cases (like debugging)
>>> should be able to know already what they want to run.
>>>
>>>
>>> Results in CI: CI will show results for both the container, and the
>>> dynamic subtests within it. The naming is:
>>>
>>> igt at binary@subcontainer  -  the container, has as its output the whole
>>> shebang that dynamic subtests printed.
>>>
>>> igt at binary@subcontainer at dynamicname  -  a dynamic subtest appears as a
>>> separate name, WITHOUT grouping or nesting. Any relation to the
>>> container will not be linked at this point. Possibly in the future
>>> when a usability wizard figures out the best way to browse results...
>>
>> Did not get this part. Is it the first or second option? Or both? If it is
>> the second then CI has to handle dynamic subtests so why we even need the
>> complication to start with?
> 
> 
> It's both. The container will report a result (based on what the
> dynamic tests did) and there's also a result for each dynamic subtest.
> 
> The complication is needed because with dynamic subtests the execution
> entry and results reporting are two different sets, reported subtests
> being a superset of the entries.
> 
> CI will handle (knocking on wood here) dynamic results fine in its
> current state, because dynamic subtest results walk like a duck, smell
> like a duck, and look like normal subtest results.

I am confused by all this. Could we then achieve the same result by just 
adding the ability to hide subtests from test enumeration?

igt_subtest_group {
	igt_subtest("a");
}

igt_dynamic_subtest_group("group") {
	igt_subtest("b");
}

# ./test --list-subtests
a
group

# .test --list-all-subtests
a
group-b

# ./test --r group
PASS

# ./test --r group-b
PASS

Would the above satisfy CI requirements? Sounds like it would be easier 
to implement wrt needed code base changes in individual tests and 
possibly the IGT core code would not be difficult either? Just needs to 
track in what kind of subgroup it is to decide what to do when 
listing/running tests.

Regards,

Tvrtko