[Piglit] Regression with eeec306bc2197a2fc0eeb064462a1244d39c16fa "framework: don't report failed cases as PASS"
Dylan Baker
baker.dylan.c at gmail.com
Wed Aug 28 12:06:39 PDT 2013
If a test has two subtests with the same name that is a bug in the test, and
the offending subtests should be renamed.
There does seem to be a bug, but I think that the bug is actually in the json
output. We are writing a full test result for each subtest, which is
incorrect. I have sent out patches to fix this. They are titled core.py: fix
subtest json reporting.
On Monday 26 August 2013 14:50:04 Tom Stellard wrote:
> On Mon, Aug 26, 2013 at 11:59:40AM -0700, Dylan Baker wrote:
> > Tom,
> > I'm a little confused, this was the exact intention of the patch series.
>
> My understanding was that the intention was to fix incorrect reporting
> when multiple subtests had the same name. For example, with the old
> behavior, this:
>
> PIGLIT:subtest {'+neg' : 'fail'}
> PIGLIT:subtest {'+neg' : 'pass'}
>
>
> The '+neg' subtest would be displayed as 'pass' in the summary, because
> the last subtest result was overwriting the first. I thought the patch
> would fix this, so that '+neg' would display as 'fail' in the summary.
>
> I think the patches fixes this, but it also introduces another problem
> with results like this:
>
> PIGLIT:subtest {'-pos' : 'pass'}
> PIGLIT:subtest {'+inf' : 'fail'}
>
> In this case the summary shows both '-pos' and '+inf' as failed, which
> it shouldn't. If this really was the intention of the patch, then I
> think it should be reverted, because it completed defeats the purpose
> of having subtests.
>
> -Tom
>
> > Dylan
> >
> > On Monday 26 August 2013 10:09:10 Tom Stellard wrote:
> > > Hi Homer,
> > >
> > > This commit introduces a regression in piglit for reporting subtest
> > > results. Now if one subtests fails, all subtests report failure, which
> > > is incorrect. I've looked through the changes in this commit, and I
> > > don't understand which change introduced the regression, could you take
> > > a look?
> > >
> > > Thanks,
> > > Tom
> > > _______________________________________________
> > > Piglit mailing list
> > > Piglit at lists.freedesktop.org
> > > http://lists.freedesktop.org/mailman/listinfo/piglit
More information about the Piglit
mailing list