[Piglit] [PATCH 3/6] dmesg: setup dmesg to be able to mark a test warn or fail

[Marek Olšák]

I'll gladly type --dmesg=radeon.

I have my own script for running piglit anyway, which calls piglit-run
and piglit-summary-html, and the result and summary directories are
named $(date)_$(time)_$(GLrenderer)_$(test_filters), and it can do a
couple of other things.

Marek

On Fri, Nov 15, 2013 at 5:26 PM, Dylan Baker <baker.dylan.c at gmail.com> wrote:
> On Friday, November 15, 2013 10:42:11 AM Daniel Vetter wrote:
>> On Thu, Nov 14, 2013 at 10:35:03AM -0800, Dylan Baker wrote:
>> > On Thursday, November 14, 2013 05:34:38 PM Daniel Vetter wrote:
>> > > On Thu, Nov 14, 2013 at 08:04:20AM -0800, Dylan Baker wrote:
>> > > > On Thursday, November 14, 2013 02:52:41 PM Daniel Vetter wrote:
>> > > > > On Tue, Nov 12, 2013 at 07:54:01AM -0800, Dylan Baker wrote:
>> > > > > > This gives the dmesg class lists of statuses that will make a test
>> > > > > > a
>> > > > > > warn or a fail, it includes a few basic checks, namely i915 errors
>> > > > > > and
>> > > > > > that tests have not segfaulted.
>> > > > > >
>> > > > > > Signed-off-by: Dylan Baker <baker.dylan.c at gmail.com>
>> > > > > > ---
>> > > > > >
>> > > > > >  framework/dmesg.py    | 36 ++++++++++++++++++++++++++++++++----
>> > > > > >  framework/exectest.py | 22 +++++++++++++++-------
>> > > > > >  2 files changed, 47 insertions(+), 11 deletions(-)
>> > > > > >
>> > > > > > diff --git a/framework/dmesg.py b/framework/dmesg.py
>> > > > > > index 9a23c14..edbea88 100644
>> > > > > > --- a/framework/dmesg.py
>> > > > > > +++ b/framework/dmesg.py
>> > > > > > @@ -22,6 +22,7 @@
>> > > > > >
>> > > > > >  """ Module implementing classes for reading posix dmesg """
>> > > > > >
>> > > > > >  import os
>> > > > > >
>> > > > > > +import re
>> > > > > >
>> > > > > >  import subprocess
>> > > > > >  from threads import synchronized_self
>> > > > > >
>> > > > > > @@ -29,8 +30,10 @@ __all__ = ['Dmesg']
>> > > > > >
>> > > > > >  # plain text list of statuses to be considered either a warn or a
>> > > > > >  fail,
>> > > > > >  any
>> > > > > >  # statuses not on this list will simply be ignored.
>> > > > > >
>> > > > > > -WARN_STATUSES = []
>> > > > > > -FAIL_STATUSES = []
>> > > > > > +WARN_STATUSES = ['segfault']
>> > > > > > +FAIL_STATUSES = ['\[drm:.*\] \*ERROR\*',
>> > > > > > +                 '\[drm\] stuck on [a-zA-Z]* ring',
>> > > > > > +                 '\[drm\] GPU crash dump saved']
>> > > > >
>> > > > > I think now that we filter out all the info/debug noise maybe we
>> > > > > could
>> > > > > go
>> > > > > the other direction and blacklist a few of the remaining things from
>> > > > > the
>> > > > > core kernel we don't care about. E.g.
>> > > > >
>> > > > > [ 3867.022895] gem_evict_every (2671) used greatest stack depth:
>> > > > > 2216
>> > > > > bytes
>> > > > > left
>> > > > >
>> > > > > is a warn level message, but I don't care one bit about it (as long
>> > > > > as
>> > > > > it
>> > > > > doesn't approach 0). But there's other warn level stuff which is
>> > > > > fairly
>> > > > > interesting.
>> > > > >
>> > > > > Just something to throw out there, I'm not sure what the best way
>> > > > > would
>> > > > > be
>> > > > > to integrate dmesg reporting for piglit in general.
>> > > > > -Daniel
>> > > >
>> > > > My personal problem with the dmesg code we have now (and with *just*
>> > > > blacklisting) is that I have an alps touchpad, it spams dmesg about 10
>> > > > times a minute, so I can't use dmesg reporting because of the massive
>> > > > number of false positives; we could use some combination of
>> > > > blacklisting
>> > > > and whitelisting however.
>> > >
>> > > That sounds like we need a piglit cmdline option to supply a regex to
>> > > filter out crap like that ... Or is the alps touchpad driver so bad
>> > > there's not even a regex we could match it all against?
>> > > -Daniel
>> >
>> > My concern is more that trying to filter out things we don't want seems
>> > like an uphill battle that will become expensive quickly. First it's my
>> > touchpad, then it's so and so's usb, and so on. I'm giong to ask some of
>> > the mesa guys here in the office to weigh in with their thoughts, since
>> > they're all around today, snice I'm basing interest on one developer's
>> > opinions.
>>
>> That's kinda what the cmdline would be for, to get rid of machine-specific
>> stuff like your touchpad. I'm retesting igt on all the machines I have
>> here right now, and thus far the stack usage warning is the only offender
>> I've seend which wasn't a genuine issue. But yeah, more input should
>> definitely help.
>> -Daniel
>
> So if I'm understanding correctly, you're asking users to craft regex on the
> commandline every time they run piglit? that sounds like a nightmare.