[Piglit] Crashing tests are not shown in the HTML summary

[Dylan Baker]

Actually, branches do have a pass fail status, it's the worst status of that 
branch's children.

I've been thinking of using a Tree structure to represent the tests in 
Summary. The problem with the current architecture (as it pertains to this 
problem) is that the the statuses of each group are auto-generated from the 
'worst' status in their children, and the only way to set a group status is 
after the entire tree has been generated.

If you write a patch to add a fake subtest I'll be sure to look at it.

On Monday, April 07, 2014 06:44:04 PM Ilia Mirkin wrote:
> Before rewriting (fun though it is!), we should figure out what we'd
> want to see, in a perfect world. Even that much isn't clear to me. Do
> we want to promote test tree structure to first-class status, thus
> being able to accurate express the situation? That would require HTML
> changes as well, I think, since only leaves can have pass/fail status
> right now, and everything else is just a n/m number. My hack should
> make it clear what happened while also minimizing the amount of
> change. I don't think that this is a common scenario (passing subtests
> followed by crash), so just making it visible may be good enough.
> 
>   -ilia
> 
> On Mon, Apr 7, 2014 at 6:34 PM, Dylan Baker <baker.dylan.c at gmail.com> wrote:
> > Ilia,
> > 
> > After looking at the code and what you said I don't think this can be
> > fixed
> > without your hack or a complete re-architecture of the summary code. I've
> > been itching for an excuse to rewrite that code again (what I wrote was
> > better than what was there before, which should say something about what
> > was there before)
> > 
> > Does that sound right to you?
> > 
> > On Monday, April 07, 2014 09:34:13 AM Ilia Mirkin wrote:
> >> This took me a little bit to figure out what was going on... so to
> >> save someone else some time: the problem is that the summary logic
> >> branches into two -- individual results, and subtest results. For the
> >> subtest results, the "overall" test result isn't kept anywhere. So the
> >> fact that the test crashed is completely lost, since the individual
> >> subtests completed.
> >> 
> >> The reported data is accurate -- those subtests did all pass; however
> >> it's incomplete. Perhaps if the test result is a crash, and there are
> >> subtests, then create a new "fake" subtest that shows that the overall
> >> test actually crashed?
> >> 
> >>   -ilia
> >> 
> >> On Mon, Apr 7, 2014 at 7:27 AM, Marek Olšák <maraeo at gmail.com> wrote:
> >> > Hi,
> >> > 
> >> > There is a serious issue with piglit that crashing tests just
> >> > disappear from the HTML summary. Let's say I apply this:
> >> > 
> >> > diff --git a/tests/fbo/fbo-generatemipmap-formats.c
> >> > b/tests/fbo/fbo-generatemipmap-formats.c
> >> > index 666030f..0d4a06b 100644
> >> > --- a/tests/fbo/fbo-generatemipmap-formats.c
> >> > +++ b/tests/fbo/fbo-generatemipmap-formats.c
> >> > @@ -431,6 +431,8 @@ test_format(const struct format_desc *format,
> >> > GLenum basetype)
> >> > 
> >> >         GLboolean pass = GL_TRUE;
> >> > 
> >> >         printf("Testing %s%s\n", format->name, tex_width == 256 ? "" :
> >> > " (NPOT)");
> >> > +        if (format->internalformat == GL_RG16)
> >> > +           *(int*)NULL = 0;
> >> > 
> >> >         tex = create_tex(format->internalformat,
> >> >         format->base_internal_format,
> >> >         
> >> >                          basetype);
> >> > 
> >> > Then, the output of the test is:
> >> > 
> >> > $ bin/fbo-generatemipmap-formats GL_ARB_texture_rg -auto
> >> > Using test set: GL_ARB_texture_rg
> >> > Testing GL_R8
> >> > PIGLIT:subtest {'GL_R8' : 'pass'}
> >> > Testing GL_R16
> >> > PIGLIT:subtest {'GL_R16' : 'pass'}
> >> > Testing GL_RG
> >> > PIGLIT:subtest {'GL_RG' : 'pass'}
> >> > Testing GL_RG8
> >> > PIGLIT:subtest {'GL_RG8' : 'pass'}
> >> > Testing GL_RG16
> >> > Segmentation fault (core dumped)
> >> > 
> >> > Let's run piglit-run.py with "-t
> >> > ARB_texture_rg/fbo-generatemipmap-formats$ -v":
> >> > 
> >> > running :: spec/ARB_texture_rg/fbo-generatemipmap-formats
> >> > crash :: spec/ARB_texture_rg/fbo-generatemipmap-formats
> >> > [1/1] crash: 1
> >> > 
> >> > Looks good, right? But there is no crash in the HTML summary:
> >> > 
> >> > http://people.freedesktop.org/~mareko/summary/
> >> > 
> >> > Do you have any idea how to fix this?
> >> > 
> >> > Marek
> >> > _______________________________________________
> >> > Piglit mailing list
> >> > Piglit at lists.freedesktop.org
> >> > http://lists.freedesktop.org/mailman/listinfo/piglit
> >> 
> >> _______________________________________________
> >> Piglit mailing list
> >> Piglit at lists.freedesktop.org
> >> http://lists.freedesktop.org/mailman/listinfo/piglit
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.freedesktop.org/archives/piglit/attachments/20140407/e7927c95/attachment.sig>