[Piglit] Crashing tests are not shown in the HTML summary

[Ilia Mirkin]

This took me a little bit to figure out what was going on... so to
save someone else some time: the problem is that the summary logic
branches into two -- individual results, and subtest results. For the
subtest results, the "overall" test result isn't kept anywhere. So the
fact that the test crashed is completely lost, since the individual
subtests completed.

The reported data is accurate -- those subtests did all pass; however
it's incomplete. Perhaps if the test result is a crash, and there are
subtests, then create a new "fake" subtest that shows that the overall
test actually crashed?

  -ilia

On Mon, Apr 7, 2014 at 7:27 AM, Marek Olšák <maraeo at gmail.com> wrote:
> Hi,
>
> There is a serious issue with piglit that crashing tests just
> disappear from the HTML summary. Let's say I apply this:
>
> diff --git a/tests/fbo/fbo-generatemipmap-formats.c
> b/tests/fbo/fbo-generatemipmap-formats.c
> index 666030f..0d4a06b 100644
> --- a/tests/fbo/fbo-generatemipmap-formats.c
> +++ b/tests/fbo/fbo-generatemipmap-formats.c
> @@ -431,6 +431,8 @@ test_format(const struct format_desc *format,
> GLenum basetype)
>         GLboolean pass = GL_TRUE;
>
>         printf("Testing %s%s\n", format->name, tex_width == 256 ? "" :
> " (NPOT)");
> +        if (format->internalformat == GL_RG16)
> +           *(int*)NULL = 0;
>         tex = create_tex(format->internalformat, format->base_internal_format,
>                          basetype);
>
> Then, the output of the test is:
>
> $ bin/fbo-generatemipmap-formats GL_ARB_texture_rg -auto
> Using test set: GL_ARB_texture_rg
> Testing GL_R8
> PIGLIT:subtest {'GL_R8' : 'pass'}
> Testing GL_R16
> PIGLIT:subtest {'GL_R16' : 'pass'}
> Testing GL_RG
> PIGLIT:subtest {'GL_RG' : 'pass'}
> Testing GL_RG8
> PIGLIT:subtest {'GL_RG8' : 'pass'}
> Testing GL_RG16
> Segmentation fault (core dumped)
>
> Let's run piglit-run.py with "-t ARB_texture_rg/fbo-generatemipmap-formats$ -v":
>
> running :: spec/ARB_texture_rg/fbo-generatemipmap-formats
> crash :: spec/ARB_texture_rg/fbo-generatemipmap-formats
> [1/1] crash: 1
>
> Looks good, right? But there is no crash in the HTML summary:
>
> http://people.freedesktop.org/~mareko/summary/
>
> Do you have any idea how to fix this?
>
> Marek
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/piglit