[Piglit] [PATCH] Ignore piglit warning status in JUnit

[Mark Janes]

Ilia Mirkin <imirkin at alum.mit.edu> writes:

> So a test that otherwise passes (i.e. has a "pass" result code) but
> generates errors in dmesg will get converted to a "dmesg-warn" code.
> You were previously treating those as failures, but now will treat
> them as success.

Yes.  I used the comment in framework/status.py to determine what
severities of piglit status correspond to JUnit failures:

    Status ordering from best to worst:
    
    pass
    dmesg-warn
    warn    
    dmesg-fail
    fail
    timeout
    crash

If warn doesn't trigger a failure, then dmesg-warn shouldn't either.
>
> I don't feel strongly either way, but just wanted to point it out for
> your consideration. Also adding Jose who IIRC is also a junit user.
> [I, btw, am not.]

Thanks, I should have remembered to do that.  In my own tests, I found
the following tests that were warning:

HSW, IVB, SNB, BDW:
 ext_transform_feedback.tessellation quads flat_last
 ext_transform_feedback.tessellation quad_strip flat_last
 ext_transform_feedback.tessellation polygon flat_last

The tests that provided output indicated pixel color accuracy was beyond
the warning threshold.

G45, G965:
 !opengl 1_1.teximage-colors gl_r32f
 !opengl 1_1.teximage-colors gl_rg32f
 !opengl 1_1.teximage-colors gl_rgb32f
 !opengl 1_1.teximage-colors gl_rgba32f

No output was provided for these warnings, so they may have have
previously been dmesg-warn.

>
> On Fri, Jun 26, 2015 at 1:40 PM, Mark Janes <mark.a.janes at intel.com> wrote:
>> JUnit has no concept of "warning".  It supports the following
>> statuses:
>>
>>  - skip
>>  - success
>>  - fail
>>  - error
>>
>> dEQP has been found to intermittently emit warnings for passed tests,
>> and this status is accurately represented in piglit json.  However,
>> current JUnit transforms them into failures.
>>
>> A test which emits a warning is more accurately represented as
>> "success" in JUnit.
>> ---
>>  framework/backends/junit.py | 3 +--
>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>
>> diff --git a/framework/backends/junit.py b/framework/backends/junit.py
>> index 632e516..7499829 100644
>> --- a/framework/backends/junit.py
>> +++ b/framework/backends/junit.py
>> @@ -142,8 +142,7 @@ class JUnitBackend(FileBackend):
>>              if data['result'] == 'skip':
>>                  res = etree.SubElement(element, 'skipped')
>>
>> -            elif data['result'] in ['warn', 'fail', 'dmesg-warn',
>> -                                    'dmesg-fail']:
>> +            elif data['result'] in ['fail', 'dmesg-fail']:
>>                  if expected_result == "failure":
>>                      err.text += "\n\nWARN: passing test as an expected failure"
>>                      res = etree.SubElement(element, 'skipped',
>> --
>> 2.1.4
>>