[Piglit] [PATCH 2/2] framework/results: Fix subtests masking crashes

[Ilia Mirkin]

On Tue, May 10, 2016 at 8:05 PM, Dylan Baker <dylan at pnwbakers.com> wrote:
> Quoting Ilia Mirkin (2016-05-10 14:32:59)
>> On Tue, May 10, 2016 at 5:32 PM, Dylan Baker <dylan at pnwbakers.com> wrote:
>> > Quoting Ilia Mirkin (2016-05-10 06:41:33)
>> >> Shouldn't the test's result just be included into the "max" no matter what?
>> >>
>> >
>> > Well, the problem is that a lot of tests were retrofitted with subtests,
>> > and now the overall status is bunk, and for new tests we generally don't
>> > add an overall result when there's subtests.
>>
>> It's surprising if the status of a test says fail but piglit-summary
>> reports it as pass. In the case where one of the subtests fails but
>> the overall status is pass, this would still say fail, no?
>>
>> What situation are you protecting against by only incorporating the
>> overall status on crash?
>
> I have this gut feeling that says skip or notrun might play havok here.
> But I guess I should look test it and see if that's the case.
>
> The other thing I guess is that there's only about 100 tests (or, c/c++
> files, really) using subtests. So maybe it would just be easier to audit
> them to make sure that they do the right thing and just add the overall
> result to the max call.

I dunno... I think in at least some cases some "stuff" happens between
subtests that can fail. I don't think it's correct to look at a test
as a shell for a bunch of disconnected subtests, and thus anything the
shell itself does is meaningless. The shell can also fail. Feel free
to audit everything :)

  -ilia