[Piglit] [PATCH 2/2] framework/results: Fix subtests masking crashes

[Dylan Baker]

Quoting Ilia Mirkin (2016-05-10 17:45:49)
> On Tue, May 10, 2016 at 8:05 PM, Dylan Baker <dylan at pnwbakers.com> wrote:
> > Quoting Ilia Mirkin (2016-05-10 14:32:59)
> >> On Tue, May 10, 2016 at 5:32 PM, Dylan Baker <dylan at pnwbakers.com> wrote:
> >> > Quoting Ilia Mirkin (2016-05-10 06:41:33)
> >> >> Shouldn't the test's result just be included into the "max" no matter what?
> >> >>
> >> >
> >> > Well, the problem is that a lot of tests were retrofitted with subtests,
> >> > and now the overall status is bunk, and for new tests we generally don't
> >> > add an overall result when there's subtests.
> >>
> >> It's surprising if the status of a test says fail but piglit-summary
> >> reports it as pass. In the case where one of the subtests fails but
> >> the overall status is pass, this would still say fail, no?
> >>
> >> What situation are you protecting against by only incorporating the
> >> overall status on crash?
> >
> > I have this gut feeling that says skip or notrun might play havok here.
> > But I guess I should look test it and see if that's the case.
> >
> > The other thing I guess is that there's only about 100 tests (or, c/c++
> > files, really) using subtests. So maybe it would just be easier to audit
> > them to make sure that they do the right thing and just add the overall
> > result to the max call.
> 
> I dunno... I think in at least some cases some "stuff" happens between
> subtests that can fail. I don't think it's correct to look at a test
> as a shell for a bunch of disconnected subtests, and thus anything the
> shell itself does is meaningless. The shell can also fail. Feel free
> to audit everything :)
> 
>   -ilia

You know. Thinking back there was a point where the framework defaulted
to FAIL instead of NOTRUN. This might be an artifact of that time.
Either way I guess just seeing what happens is the easiest way to figure
out if it's going to work or not
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: signature
URL: <https://lists.freedesktop.org/archives/piglit/attachments/20160510/ad26458e/attachment.sig>