[Piglit] [PATCH] Ignore piglit warning status in JUnit

[Ilia Mirkin]

On Fri, Jun 26, 2015 at 2:11 PM, Mark Janes <mark.a.janes at intel.com> wrote:
> Ilia Mirkin <imirkin at alum.mit.edu> writes:
>
>> So a test that otherwise passes (i.e. has a "pass" result code) but
>> generates errors in dmesg will get converted to a "dmesg-warn" code.
>> You were previously treating those as failures, but now will treat
>> them as success.
>
> Yes.  I used the comment in framework/status.py to determine what
> severities of piglit status correspond to JUnit failures:
>
>     Status ordering from best to worst:
>
>     pass
>     dmesg-warn
>     warn
>     dmesg-fail
>     fail
>     timeout
>     crash

Pretty sure that dmesg-warn is worse than warn, and quite sure that
dmesg-fail is worse than fail. I think that comment is out of date.
Dylan?

>
> If warn doesn't trigger a failure, then dmesg-warn shouldn't either.
>>
>> I don't feel strongly either way, but just wanted to point it out for
>> your consideration. Also adding Jose who IIRC is also a junit user.
>> [I, btw, am not.]
>
> Thanks, I should have remembered to do that.  In my own tests, I found
> the following tests that were warning:
>
> HSW, IVB, SNB, BDW:
>  ext_transform_feedback.tessellation quads flat_last
>  ext_transform_feedback.tessellation quad_strip flat_last
>  ext_transform_feedback.tessellation polygon flat_last

Yep, the test verifies something questionable and returns a 'warn'
when that questionable thing is not satisifed. (The order of
tessellation of complex primitives when doing transform feedback is
unspecified.)

>
> The tests that provided output indicated pixel color accuracy was beyond
> the warning threshold.
>
> G45, G965:
>  !opengl 1_1.teximage-colors gl_r32f
>  !opengl 1_1.teximage-colors gl_rg32f
>  !opengl 1_1.teximage-colors gl_rgb32f
>  !opengl 1_1.teximage-colors gl_rgba32f
>
> No output was provided for these warnings, so they may have have
> previously been dmesg-warn.
>
>>
>> On Fri, Jun 26, 2015 at 1:40 PM, Mark Janes <mark.a.janes at intel.com> wrote:
>>> JUnit has no concept of "warning".  It supports the following
>>> statuses:
>>>
>>>  - skip
>>>  - success
>>>  - fail
>>>  - error
>>>
>>> dEQP has been found to intermittently emit warnings for passed tests,
>>> and this status is accurately represented in piglit json.  However,
>>> current JUnit transforms them into failures.
>>>
>>> A test which emits a warning is more accurately represented as
>>> "success" in JUnit.
>>> ---
>>>  framework/backends/junit.py | 3 +--
>>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>>
>>> diff --git a/framework/backends/junit.py b/framework/backends/junit.py
>>> index 632e516..7499829 100644
>>> --- a/framework/backends/junit.py
>>> +++ b/framework/backends/junit.py
>>> @@ -142,8 +142,7 @@ class JUnitBackend(FileBackend):
>>>              if data['result'] == 'skip':
>>>                  res = etree.SubElement(element, 'skipped')
>>>
>>> -            elif data['result'] in ['warn', 'fail', 'dmesg-warn',
>>> -                                    'dmesg-fail']:
>>> +            elif data['result'] in ['fail', 'dmesg-fail']:
>>>                  if expected_result == "failure":
>>>                      err.text += "\n\nWARN: passing test as an expected failure"
>>>                      res = etree.SubElement(element, 'skipped',
>>> --
>>> 2.1.4
>>>