[Piglit] [PATCH 2/2] framework/results: Fix subtests masking crashes

[Brian Paul]

On 05/09/2016 03:46 PM, Dylan Baker wrote:
> In piglit when a test generates subtests we treat the "worst" subtest as
> the test status, and in most cases this works as expected. There is at
> least one case where this is not correct, and that is the case of crash.
>
> There are a couple of reasons that crash should not be masked. One is
> that it is generated by the framework when the test binary hits an
> assert or segfaults, or any number of similar cases. The second is that
> it may mean that all of the subtests did not run, as such we don't want
> the status to be masked by the "worst" subtest which would be fail at
> worst.
>
> Signed-off-by: Dylan Baker <dylanx.c.baker at intel.com>
> ---
>
>   framework/results.py | 7 +++++--
>   1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/framework/results.py b/framework/results.py
> index 469abeb..2095d90 100644
> --- a/framework/results.py
> +++ b/framework/results.py
> @@ -173,10 +173,13 @@ class TestResult(object):
>           """Return the result of the test.
>
>           If there are subtests return the "worst" value of those subtests. If
> -        there are not return the stored value of the test.
> +        there are not return the stored value of the test. There is an
> +        exception to this rule, and that's if the status is crash; since this
> +        status is set by the framework, and can be generated even when some or
> +        all unit tests pass.
>
>           """
> -        if self.subtests:
> +        if self.subtests and self.__result != status.CRASH:
>               return max(six.itervalues(self.subtests))
>           return self.__result
>
>

This looks OK to me, and it fixes the incorrect "pass" result in the 
json file.

Reviewed-by: Brian Paul <brianp at vmware.com>
Tested-by: Brian Paul <brianp at vmware.com>

However, when I run piglit-summary.py -s on the results file I get:

        name:    new
        ----  ------
        pass:     70
        fail:      2
       crash:      0
        skip:     19
     timeout:      0
        warn:      0
  incomplete:      0
  dmesg-warn:      0
  dmesg-fail:      0
     changes:      0
       fixes:      0
regressions:      0
       total:     91

The "crash" result in the json file doesn't show up here.  I believe 
when there's subtests, we're counting the subtest results but not the 
overall test result.  Is there a way to make sure that the crash result 
shows up in the summary?

-Brian