[Piglit] [PATCH 2/2] framework/test/base.py: Handle fail cases for tests.

[Mark Janes]

Reviewed-by: Mark Janes <mark.a.janes at intel.com>

baker.dylan.c at gmail.com writes:

> From: Dylan Baker <baker.dylan.c at gmail.com>
>
> I'm going to admit I'm a bit puzzled how this could have slipped through
> without being caught (I'm guessing an unrelated change uncovered this).
> But basically if a test doesn't raise an exception, and the returncode
> is > 0, it should mark the test with a status of "fail", but it doesn't.
> Instead the default status "notrun" is passed to the logger, which it
> doesn't support, and an exception is raised.
>
> This patch corrects that problem.
>
> bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=93340
> cc: mark.a.janes at intel.com
> Signed-off-by: Dylan Baker <dylanx.c.baker at intel.com>
> ---
>
> I'm not really sure this is exactly the right solution. I wonder if we'd
> be better just to get rid of the "warn" and just say if the returncode
> is 0 leave it alone, if it's crash/timeout (as defined by the function),
> do that, otherwise mark it as "fail".
>
>  framework/test/base.py        |  7 +++++--
>  framework/tests/base_tests.py | 15 +++++++++++++++
>  2 files changed, 20 insertions(+), 2 deletions(-)
>
> diff --git a/framework/test/base.py b/framework/test/base.py
> index 378f185..4232e6c 100644
> --- a/framework/test/base.py
> +++ b/framework/test/base.py
> @@ -210,8 +210,11 @@ class Test(object):
>                  self.result.result = 'timeout'
>              else:
>                  self.result.result = 'crash'
> -        elif self.result.returncode != 0 and self.result.result == 'pass':
> -            self.result.result = 'warn'
> +        elif self.result.returncode != 0:
> +            if self.result.result == 'pass':
> +                self.result.result = 'warn'
> +            else:
> +                self.result.result = 'fail'
>  
>      def run(self):
>          """
> diff --git a/framework/tests/base_tests.py b/framework/tests/base_tests.py
> index c005273..20bea87 100644
> --- a/framework/tests/base_tests.py
> +++ b/framework/tests/base_tests.py
> @@ -260,3 +260,18 @@ class TestValgrindMixinRun(object):
>          test.result.returncode = 1
>          test.run()
>          nt.eq_(test.result.result, 'fail')
> +
> +
> +def test_interpret_result_greater_zero():
> +    """test.base.Test.interpret_result: A test with status > 0 is fail"""
> +    class _Test(Test):
> +        def interpret_result(self):
> +            super(_Test, self).interpret_result()
> +
> +    test = _Test(['foobar'])
> +    test.result.returncode = 1
> +    test.result.out = 'this is some\nstdout'
> +    test.result.err = 'this is some\nerrors'
> +    test.interpret_result()
> +
> +    nt.eq_(test.result.result, 'fail')
> -- 
> 2.6.4