[Piglit] Crashing tests are not shown in the HTML summary

[Dylan Baker]

Ilia,

After looking at the code and what you said I don't think this can be fixed 
without your hack or a complete re-architecture of the summary code. I've been 
itching for an excuse to rewrite that code again (what I wrote was better than 
what was there before, which should say something about what was there before)

Does that sound right to you?

On Monday, April 07, 2014 09:34:13 AM Ilia Mirkin wrote:
> This took me a little bit to figure out what was going on... so to
> save someone else some time: the problem is that the summary logic
> branches into two -- individual results, and subtest results. For the
> subtest results, the "overall" test result isn't kept anywhere. So the
> fact that the test crashed is completely lost, since the individual
> subtests completed.
> 
> The reported data is accurate -- those subtests did all pass; however
> it's incomplete. Perhaps if the test result is a crash, and there are
> subtests, then create a new "fake" subtest that shows that the overall
> test actually crashed?
> 
>   -ilia
> 
> On Mon, Apr 7, 2014 at 7:27 AM, Marek Olšák <maraeo at gmail.com> wrote:
> > Hi,
> > 
> > There is a serious issue with piglit that crashing tests just
> > disappear from the HTML summary. Let's say I apply this:
> > 
> > diff --git a/tests/fbo/fbo-generatemipmap-formats.c
> > b/tests/fbo/fbo-generatemipmap-formats.c
> > index 666030f..0d4a06b 100644
> > --- a/tests/fbo/fbo-generatemipmap-formats.c
> > +++ b/tests/fbo/fbo-generatemipmap-formats.c
> > @@ -431,6 +431,8 @@ test_format(const struct format_desc *format,
> > GLenum basetype)
> > 
> >         GLboolean pass = GL_TRUE;
> > 
> >         printf("Testing %s%s\n", format->name, tex_width == 256 ? "" :
> > " (NPOT)");
> > +        if (format->internalformat == GL_RG16)
> > +           *(int*)NULL = 0;
> > 
> >         tex = create_tex(format->internalformat,
> >         format->base_internal_format,
> >         
> >                          basetype);
> > 
> > Then, the output of the test is:
> > 
> > $ bin/fbo-generatemipmap-formats GL_ARB_texture_rg -auto
> > Using test set: GL_ARB_texture_rg
> > Testing GL_R8
> > PIGLIT:subtest {'GL_R8' : 'pass'}
> > Testing GL_R16
> > PIGLIT:subtest {'GL_R16' : 'pass'}
> > Testing GL_RG
> > PIGLIT:subtest {'GL_RG' : 'pass'}
> > Testing GL_RG8
> > PIGLIT:subtest {'GL_RG8' : 'pass'}
> > Testing GL_RG16
> > Segmentation fault (core dumped)
> > 
> > Let's run piglit-run.py with "-t
> > ARB_texture_rg/fbo-generatemipmap-formats$ -v":
> > 
> > running :: spec/ARB_texture_rg/fbo-generatemipmap-formats
> > crash :: spec/ARB_texture_rg/fbo-generatemipmap-formats
> > [1/1] crash: 1
> > 
> > Looks good, right? But there is no crash in the HTML summary:
> > 
> > http://people.freedesktop.org/~mareko/summary/
> > 
> > Do you have any idea how to fix this?
> > 
> > Marek
> > _______________________________________________
> > Piglit mailing list
> > Piglit at lists.freedesktop.org
> > http://lists.freedesktop.org/mailman/listinfo/piglit
> 
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/piglit
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.freedesktop.org/archives/piglit/attachments/20140407/bdc26c8e/attachment-0001.sig>