[Piglit] [PATCH 2/2] framework/results: Fix subtests masking crashes

[Dylan Baker]

Quoting Ilia Mirkin (2016-05-10 06:41:33)
> Shouldn't the test's result just be included into the "max" no matter what?
> 
> On Mon, May 9, 2016 at 5:46 PM, Dylan Baker <dylan at pnwbakers.com> wrote:
> > In piglit when a test generates subtests we treat the "worst" subtest as
> > the test status, and in most cases this works as expected. There is at
> > least one case where this is not correct, and that is the case of crash.
> >
> > There are a couple of reasons that crash should not be masked. One is
> > that it is generated by the framework when the test binary hits an
> > assert or segfaults, or any number of similar cases. The second is that
> > it may mean that all of the subtests did not run, as such we don't want
> > the status to be masked by the "worst" subtest which would be fail at
> > worst.
> >
> > Signed-off-by: Dylan Baker <dylanx.c.baker at intel.com>
> > ---
> >
> >  framework/results.py | 7 +++++--
> >  1 file changed, 5 insertions(+), 2 deletions(-)
> >
> > diff --git a/framework/results.py b/framework/results.py
> > index 469abeb..2095d90 100644
> > --- a/framework/results.py
> > +++ b/framework/results.py
> > @@ -173,10 +173,13 @@ class TestResult(object):
> >          """Return the result of the test.
> >
> >          If there are subtests return the "worst" value of those subtests. If
> > -        there are not return the stored value of the test.
> > +        there are not return the stored value of the test. There is an
> > +        exception to this rule, and that's if the status is crash; since this
> > +        status is set by the framework, and can be generated even when some or
> > +        all unit tests pass.
> >
> >          """
> > -        if self.subtests:
> > +        if self.subtests and self.__result != status.CRASH:
> >              return max(six.itervalues(self.subtests))
> >          return self.__result
> >
> > --
> > 2.8.2
> >
> > _______________________________________________
> > Piglit mailing list
> > Piglit at lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/piglit

Well, the problem is that a lot of tests were retrofitted with subtests,
and now the overall status is bunk, and for new tests we generally don't
add an overall result when there's subtests.

Dylan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: signature
URL: <https://lists.freedesktop.org/archives/piglit/attachments/20160510/3754f8aa/attachment.sig>