[Piglit] [Intel-gfx] [PATCH] tests/igt: dmesg noise is a kernel failure

[Daniel Vetter]

On Thu, Oct 6, 2016 at 11:16 AM, Chris Wilson <chris at chris-wilson.co.uk> wrote:
> On Thu, Oct 06, 2016 at 11:07:17AM +0200, Daniel Vetter wrote:
>> At least when testing the kernel. In normal programs pretty much all
>> the dmesg noise would simply be replaced by debug asserts, but in the
>> kernel we try rely hard to not fall over minor inconsistencies.
>>
>> Still for CI purposes there's not really a difference, hence don't
>> treat it as such.
>>
>> Motivated since once again I've seen a statistics where this was split
>> up, and then a reduction of "failures" (but in reality just trading
>> them in for more "warnings") praised as success.
>
> Hear, hear! dfail == fail, and dwarn == warn.
>
> Calling a failure, a dfail just softens that it failed.

Maybe a bit more motiviation why I think all dmesg noise should be
tracked as a full-on failure. In userspace there's tons of consistency
checks, and those are done using assert(). Debug builds have them,
production ones dont, and if you hit it your app dies, and it's
counted as a failure under CI.

In the kernel it's not great if we just die, so we try very hard to
limp along in all these cases instead of dying. But it's still a
consistency check firing, and imo should be treated as such. "warn"
would then just be for for igt self-checks in the test
library/infrastructure itself.

The other reasons is CI: "warn" essentially means "something might be
wrong, a human needs to check". That's not all that useful for
automating stuff, hence why I think we really shouldn't have that
status, at least as a usual one. If we go with relabelling, then a
consequence is also that we might need to remove some more
questionable in-kernel checks.

Re Ville's concern: I don't mind whether we call it dmesg-fail or fail
in the end. If it helps folks to scan through results I'm fine with
changing it to dmesg-fail. But dmesg-warn imo needs to go (for igt). I
think dmesg-warn is the right choice for everything else, where some
kernel noise might not always mean that there's an issue with your
userspace. So again needs a human to check what's going on.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch