[igt-dev] [PATCH i-g-t 1/1] runner: Add explicit "notrun" results for tests that were not executed

[Petri Latvala]

On Tue, Nov 13, 2018 at 01:30:26PM +0000, Chris Wilson wrote:
> Quoting Petri Latvala (2018-11-13 13:04:30)
> > When possible, all tests we know we were going to attempt to execute
> > now appear in the results as "notrun". The only known case where it's
> > not possible to add an explicit "notrun" is when running in
> > multiple-mode, because "no subtests" and "run all subtests, we didn't
> > list them beforehand" are represented the same.
> 
> Sounds strange for a runner with which we can integrate with the igt
> framework not know whether or not a test started. Although the easiest
> procedure would seem to be that all tests should include at least one
> igt_subtest().

That wouldn't work for the case I'm describing. Although it seems I
didn't describe it in enough detail. And there's a different case
lurking there too.

When running in multiple-mode, if you specified e.g. -t kms_flip, the
job list creation sees that the binary "kms_flip" is enough to match
that filter, ergo, we run all subtests of that binary, we don't need
to execute kms_flip --list-subtests to get the subtest list
explicitly. The job list will contain:

 kms_flip

If we also specified e.g. -t testdisplay, the job list will contain:

 kms_flip
 testdisplay

Now, the purpose of this patch was for Martin to know from the results
what our pass rate is, and for that he needs the list of all tests,
and we can get there by explicitly getting a "notrun" for tests that
didn't get executed (due to an incomplete). We can't know what
kms_flip subtests to get the result for, because we didn't list
them. Knowing that it _has_ subtests doesn't help.

For Intel CI, this is a moot point because we use testlists
everywhere. Even with multiple-mode, we get the full list of subtests
explicitly stated and we can fill in "notrun" for them.


-- 
Petri Latvala