[Piglit] [PATCH] Ignore piglit warning status in JUnit
imirkin at alum.mit.edu
Fri Jun 26 11:28:57 PDT 2015
On Fri, Jun 26, 2015 at 2:11 PM, Mark Janes <mark.a.janes at intel.com> wrote:
> Ilia Mirkin <imirkin at alum.mit.edu> writes:
>> So a test that otherwise passes (i.e. has a "pass" result code) but
>> generates errors in dmesg will get converted to a "dmesg-warn" code.
>> You were previously treating those as failures, but now will treat
>> them as success.
> Yes. I used the comment in framework/status.py to determine what
> severities of piglit status correspond to JUnit failures:
> Status ordering from best to worst:
Pretty sure that dmesg-warn is worse than warn, and quite sure that
dmesg-fail is worse than fail. I think that comment is out of date.
> If warn doesn't trigger a failure, then dmesg-warn shouldn't either.
>> I don't feel strongly either way, but just wanted to point it out for
>> your consideration. Also adding Jose who IIRC is also a junit user.
>> [I, btw, am not.]
> Thanks, I should have remembered to do that. In my own tests, I found
> the following tests that were warning:
> HSW, IVB, SNB, BDW:
> ext_transform_feedback.tessellation quads flat_last
> ext_transform_feedback.tessellation quad_strip flat_last
> ext_transform_feedback.tessellation polygon flat_last
Yep, the test verifies something questionable and returns a 'warn'
when that questionable thing is not satisifed. (The order of
tessellation of complex primitives when doing transform feedback is
> The tests that provided output indicated pixel color accuracy was beyond
> the warning threshold.
> G45, G965:
> !opengl 1_1.teximage-colors gl_r32f
> !opengl 1_1.teximage-colors gl_rg32f
> !opengl 1_1.teximage-colors gl_rgb32f
> !opengl 1_1.teximage-colors gl_rgba32f
> No output was provided for these warnings, so they may have have
> previously been dmesg-warn.
>> On Fri, Jun 26, 2015 at 1:40 PM, Mark Janes <mark.a.janes at intel.com> wrote:
>>> JUnit has no concept of "warning". It supports the following
>>> - skip
>>> - success
>>> - fail
>>> - error
>>> dEQP has been found to intermittently emit warnings for passed tests,
>>> and this status is accurately represented in piglit json. However,
>>> current JUnit transforms them into failures.
>>> A test which emits a warning is more accurately represented as
>>> "success" in JUnit.
>>> framework/backends/junit.py | 3 +--
>>> 1 file changed, 1 insertion(+), 2 deletions(-)
>>> diff --git a/framework/backends/junit.py b/framework/backends/junit.py
>>> index 632e516..7499829 100644
>>> --- a/framework/backends/junit.py
>>> +++ b/framework/backends/junit.py
>>> @@ -142,8 +142,7 @@ class JUnitBackend(FileBackend):
>>> if data['result'] == 'skip':
>>> res = etree.SubElement(element, 'skipped')
>>> - elif data['result'] in ['warn', 'fail', 'dmesg-warn',
>>> - 'dmesg-fail']:
>>> + elif data['result'] in ['fail', 'dmesg-fail']:
>>> if expected_result == "failure":
>>> err.text += "\n\nWARN: passing test as an expected failure"
>>> res = etree.SubElement(element, 'skipped',
More information about the Piglit