[Intel-gfx] [PATCH] tests/igt: dmesg noise is a kernel failure

[Tahvanainen, Jari]

Double checking the full picture of the impact: 
- Current - test result value list for igt (=default piglit): pass, warn, dmesg-warn, fail, dmesg-fail, timeout, crash, incomplete
- Intent with proposal below - test result value list for igt (kernel testing): pass, warn, fail, dmesg-fail, timeout, crash, incomplete
Right or wrong?
Or will fail vanish from this list too like Chris said? Or will there be scenario where one can have fail without dmesg?

BR, Jari

-----Original Message-----
From: Chris Wilson [mailto:chris at chris-wilson.co.uk] 
Sent: Friday, October 7, 2016 10:12 AM
To: Daniel Vetter <daniel.vetter at ffwll.ch>
Cc: Intel Graphics Development <intel-gfx at lists.freedesktop.org>; piglit discussion list <piglit at lists.freedesktop.org>; Ville Syrjälä <ville.syrjala at linux.intel.com>; Tahvanainen, Jari <jari.tahvanainen at intel.com>; Latvala, Petri <petri.latvala at intel.com>; Vetter, Daniel <daniel.vetter at intel.com>
Subject: Re: [PATCH] tests/igt: dmesg noise is a kernel failure

On Fri, Oct 07, 2016 at 09:06:31AM +0200, Daniel Vetter wrote:
> At least when testing the kernel. In normal programs pretty much all 
> the dmesg noise would simply be replaced by debug asserts, but in the 
> kernel we try rely hard to not fall over minor inconsistencies.
> 
> Still for CI purposes there's not really a difference, hence don't 
> treat it as such.
> 
> Motivated since once again I've seen a statistics where this was split 
> up, and then a reduction of "failures" (but in reality just trading 
> them in for more "warnings") praised as success.
> 
> v2: Clamp to "dmesg-fail" to keep dmesg noise easily identifiable 
> (Ville).
> 
> Cc: Chris Wilson <chris at chris-wilson.co.uk>
> Cc: Ville Syrjälä <ville.syrjala at linux.intel.com>
> Acked-by: Dylan Baker <dylan at pnwbakers.com>
> Cc: jari.tahvanainen at intel.com
> Cc: Petri Latvala <petri.latvala at intel.com>
> Signed-off-by: Daniel Vetter <daniel.vetter at intel.com>
> ---
>  tests/igt.py | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/tests/igt.py b/tests/igt.py index 
> 7ebb03646b50..21e55e115654 100644
> --- a/tests/igt.py
> +++ b/tests/igt.py
> @@ -123,6 +123,10 @@ class IGTTest(Test):
>          else:
>              self.result.result = 'fail'
>  
> +        # all dmesg noise is considered a test failure when testing the kernel
> +        if self.result.dmesg
> +            self.result.result = 'dmesg-fail'

This is changing a fail to dmesg-fail. I hate that.
-Chris

--
Chris Wilson, Intel Open Source Technology Centre
---------------------------------------------------------------------
Intel Finland Oy
Registered Address: PL 281, 00181 Helsinki 
Business Identity Code: 0357606 - 4 
Domiciled in Helsinki 

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.