[Intel-gfx] [PATCH] tests/igt: dmesg noise is a kernel failure

[Ville Syrjälä]

On Fri, Oct 07, 2016 at 08:12:02AM +0100, Chris Wilson wrote:
> On Fri, Oct 07, 2016 at 09:06:31AM +0200, Daniel Vetter wrote:
> > At least when testing the kernel. In normal programs pretty much all
> > the dmesg noise would simply be replaced by debug asserts, but in the
> > kernel we try rely hard to not fall over minor inconsistencies.
> > 
> > Still for CI purposes there's not really a difference, hence don't
> > treat it as such.
> > 
> > Motivated since once again I've seen a statistics where this was split
> > up, and then a reduction of "failures" (but in reality just trading
> > them in for more "warnings") praised as success.
> > 
> > v2: Clamp to "dmesg-fail" to keep dmesg noise easily identifiable
> > (Ville).
> > 
> > Cc: Chris Wilson <chris at chris-wilson.co.uk>
> > Cc: Ville Syrjälä <ville.syrjala at linux.intel.com>
> > Acked-by: Dylan Baker <dylan at pnwbakers.com>
> > Cc: jari.tahvanainen at intel.com
> > Cc: Petri Latvala <petri.latvala at intel.com>
> > Signed-off-by: Daniel Vetter <daniel.vetter at intel.com>
> > ---
> >  tests/igt.py | 4 ++++
> >  1 file changed, 4 insertions(+)
> > 
> > diff --git a/tests/igt.py b/tests/igt.py
> > index 7ebb03646b50..21e55e115654 100644
> > --- a/tests/igt.py
> > +++ b/tests/igt.py
> > @@ -123,6 +123,10 @@ class IGTTest(Test):
> >          else:
> >              self.result.result = 'fail'
> >  
> > +        # all dmesg noise is considered a test failure when testing the kernel
> > +        if self.result.dmesg
> > +            self.result.result = 'dmesg-fail'
> 
> This is changing a fail to dmesg-fail. I hate that.

I don't know if there's a way to win here. We're trying to reduce 4
states to 3 states for whatever reason, so we're going to lose
information :(

Doing it this way you'll have to go through every dmesg-fail as well as
the fails if you want to see all test failures. Doing it the other way
means going through all the fails to find the dmesg warns.

-- 
Ville Syrjälä
Intel OTC