[Piglit] Crashing tests are not shown in the HTML summary

[Ilia Mirkin]

Before rewriting (fun though it is!), we should figure out what we'd
want to see, in a perfect world. Even that much isn't clear to me. Do
we want to promote test tree structure to first-class status, thus
being able to accurate express the situation? That would require HTML
changes as well, I think, since only leaves can have pass/fail status
right now, and everything else is just a n/m number. My hack should
make it clear what happened while also minimizing the amount of
change. I don't think that this is a common scenario (passing subtests
followed by crash), so just making it visible may be good enough.

  -ilia

On Mon, Apr 7, 2014 at 6:34 PM, Dylan Baker <baker.dylan.c at gmail.com> wrote:
> Ilia,
>
> After looking at the code and what you said I don't think this can be fixed
> without your hack or a complete re-architecture of the summary code. I've been
> itching for an excuse to rewrite that code again (what I wrote was better than
> what was there before, which should say something about what was there before)
>
> Does that sound right to you?
>
> On Monday, April 07, 2014 09:34:13 AM Ilia Mirkin wrote:
>> This took me a little bit to figure out what was going on... so to
>> save someone else some time: the problem is that the summary logic
>> branches into two -- individual results, and subtest results. For the
>> subtest results, the "overall" test result isn't kept anywhere. So the
>> fact that the test crashed is completely lost, since the individual
>> subtests completed.
>>
>> The reported data is accurate -- those subtests did all pass; however
>> it's incomplete. Perhaps if the test result is a crash, and there are
>> subtests, then create a new "fake" subtest that shows that the overall
>> test actually crashed?
>>
>>   -ilia
>>
>> On Mon, Apr 7, 2014 at 7:27 AM, Marek Olšák <maraeo at gmail.com> wrote:
>> > Hi,
>> >
>> > There is a serious issue with piglit that crashing tests just
>> > disappear from the HTML summary. Let's say I apply this:
>> >
>> > diff --git a/tests/fbo/fbo-generatemipmap-formats.c
>> > b/tests/fbo/fbo-generatemipmap-formats.c
>> > index 666030f..0d4a06b 100644
>> > --- a/tests/fbo/fbo-generatemipmap-formats.c
>> > +++ b/tests/fbo/fbo-generatemipmap-formats.c
>> > @@ -431,6 +431,8 @@ test_format(const struct format_desc *format,
>> > GLenum basetype)
>> >
>> >         GLboolean pass = GL_TRUE;
>> >
>> >         printf("Testing %s%s\n", format->name, tex_width == 256 ? "" :
>> > " (NPOT)");
>> > +        if (format->internalformat == GL_RG16)
>> > +           *(int*)NULL = 0;
>> >
>> >         tex = create_tex(format->internalformat,
>> >         format->base_internal_format,
>> >
>> >                          basetype);
>> >
>> > Then, the output of the test is:
>> >
>> > $ bin/fbo-generatemipmap-formats GL_ARB_texture_rg -auto
>> > Using test set: GL_ARB_texture_rg
>> > Testing GL_R8
>> > PIGLIT:subtest {'GL_R8' : 'pass'}
>> > Testing GL_R16
>> > PIGLIT:subtest {'GL_R16' : 'pass'}
>> > Testing GL_RG
>> > PIGLIT:subtest {'GL_RG' : 'pass'}
>> > Testing GL_RG8
>> > PIGLIT:subtest {'GL_RG8' : 'pass'}
>> > Testing GL_RG16
>> > Segmentation fault (core dumped)
>> >
>> > Let's run piglit-run.py with "-t
>> > ARB_texture_rg/fbo-generatemipmap-formats$ -v":
>> >
>> > running :: spec/ARB_texture_rg/fbo-generatemipmap-formats
>> > crash :: spec/ARB_texture_rg/fbo-generatemipmap-formats
>> > [1/1] crash: 1
>> >
>> > Looks good, right? But there is no crash in the HTML summary:
>> >
>> > http://people.freedesktop.org/~mareko/summary/
>> >
>> > Do you have any idea how to fix this?
>> >
>> > Marek
>> > _______________________________________________
>> > Piglit mailing list
>> > Piglit at lists.freedesktop.org
>> > http://lists.freedesktop.org/mailman/listinfo/piglit
>>
>> _______________________________________________
>> Piglit mailing list
>> Piglit at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/piglit