[Piglit] Regression with eeec306bc2197a2fc0eeb064462a1244d39c16fa "framework: don't report failed cases as PASS"

[Dylan Baker]

If a test has two subtests with the same name that is a bug in the test, and  
the offending subtests should be renamed.

There does seem to be a bug, but I think that the bug is actually in the json 
output. We are writing a full test result for each subtest, which is 
incorrect. I have sent out patches to fix this. They are titled core.py: fix 
subtest json reporting.

On Monday 26 August 2013 14:50:04 Tom Stellard wrote:
> On Mon, Aug 26, 2013 at 11:59:40AM -0700, Dylan Baker wrote:
> > Tom,
> > I'm a little confused, this was the exact intention of the patch series.
> 
> My understanding was that the intention was to fix incorrect reporting
> when multiple subtests had the same name.  For example, with the old
> behavior, this:
> 
> PIGLIT:subtest {'+neg' : 'fail'}
> PIGLIT:subtest {'+neg' : 'pass'}
> 
> 
> The '+neg' subtest would be displayed as 'pass' in the summary, because
> the last subtest result was overwriting the first.  I thought the patch
> would fix this, so that '+neg' would display as 'fail' in the summary.
> 
> I think the patches fixes this, but it also introduces another problem
> with results like this:
> 
> PIGLIT:subtest {'-pos' : 'pass'}
> PIGLIT:subtest {'+inf' : 'fail'}
> 
> In this case the summary shows both '-pos' and '+inf' as failed, which
> it shouldn't.  If this really was the intention of the patch, then I
> think it should be reverted, because it completed defeats the purpose
> of having subtests.
> 
> -Tom
> 
> > Dylan
> > 
> > On Monday 26 August 2013 10:09:10 Tom Stellard wrote:
> > > Hi Homer,
> > > 
> > > This commit introduces a regression in piglit for reporting subtest
> > > results.  Now if one subtests fails, all subtests report failure, which
> > > is incorrect.  I've looked through the changes in this commit, and I
> > > don't understand which change introduced the regression, could you take
> > > a look?
> > > 
> > > Thanks,
> > > Tom
> > > _______________________________________________
> > > Piglit mailing list
> > > Piglit at lists.freedesktop.org
> > > http://lists.freedesktop.org/mailman/listinfo/piglit