[Piglit] [PATCH] Ignore piglit warning status in JUnit

[Jose Fonseca]

No objections from me (I have enough crashes and fails to worry about, 
so "warns" are rarely something I have time for.)

It might make sense to have an option like GCC's "-Werror" that makes 
warnings errors.  Though I suppose anybody interesting in finding out 
warnings can also run the json backend.

Jose

On 26/06/15 19:11, Mark Janes wrote:
> Ilia Mirkin <imirkin at alum.mit.edu> writes:
>
>> So a test that otherwise passes (i.e. has a "pass" result code) but
>> generates errors in dmesg will get converted to a "dmesg-warn" code.
>> You were previously treating those as failures, but now will treat
>> them as success.
>
> Yes.  I used the comment in framework/status.py to determine what
> severities of piglit status correspond to JUnit failures:
>
>      Status ordering from best to worst:
>
>      pass
>      dmesg-warn
>      warn
>      dmesg-fail
>      fail
>      timeout
>      crash
>
> If warn doesn't trigger a failure, then dmesg-warn shouldn't either.
>>
>> I don't feel strongly either way, but just wanted to point it out for
>> your consideration. Also adding Jose who IIRC is also a junit user.
>> [I, btw, am not.]
>
> Thanks, I should have remembered to do that.  In my own tests, I found
> the following tests that were warning:
>
> HSW, IVB, SNB, BDW:
>   ext_transform_feedback.tessellation quads flat_last
>   ext_transform_feedback.tessellation quad_strip flat_last
>   ext_transform_feedback.tessellation polygon flat_last
>
> The tests that provided output indicated pixel color accuracy was beyond
> the warning threshold.
>
> G45, G965:
>   !opengl 1_1.teximage-colors gl_r32f
>   !opengl 1_1.teximage-colors gl_rg32f
>   !opengl 1_1.teximage-colors gl_rgb32f
>   !opengl 1_1.teximage-colors gl_rgba32f
>
> No output was provided for these warnings, so they may have have
> previously been dmesg-warn.
>
>>
>> On Fri, Jun 26, 2015 at 1:40 PM, Mark Janes <mark.a.janes at intel.com> wrote:
>>> JUnit has no concept of "warning".  It supports the following
>>> statuses:
>>>
>>>   - skip
>>>   - success
>>>   - fail
>>>   - error
>>>
>>> dEQP has been found to intermittently emit warnings for passed tests,
>>> and this status is accurately represented in piglit json.  However,
>>> current JUnit transforms them into failures.
>>>
>>> A test which emits a warning is more accurately represented as
>>> "success" in JUnit.
>>> ---
>>>   framework/backends/junit.py | 3 +--
>>>   1 file changed, 1 insertion(+), 2 deletions(-)
>>>
>>> diff --git a/framework/backends/junit.py b/framework/backends/junit.py
>>> index 632e516..7499829 100644
>>> --- a/framework/backends/junit.py
>>> +++ b/framework/backends/junit.py
>>> @@ -142,8 +142,7 @@ class JUnitBackend(FileBackend):
>>>               if data['result'] == 'skip':
>>>                   res = etree.SubElement(element, 'skipped')
>>>
>>> -            elif data['result'] in ['warn', 'fail', 'dmesg-warn',
>>> -                                    'dmesg-fail']:
>>> +            elif data['result'] in ['fail', 'dmesg-fail']:
>>>                   if expected_result == "failure":
>>>                       err.text += "\n\nWARN: passing test as an expected failure"
>>>                       res = etree.SubElement(element, 'skipped',
>>> --
>>> 2.1.4
>>>