[Piglit] [PATCH] Ignore piglit warning status in JUnit

[Ilia Mirkin]

On Fri, Jun 26, 2015 at 2:27 PM, Daniel Vetter <daniel at ffwll.ch> wrote:
> On Fri, Jun 26, 2015 at 10:40:28AM -0700, Mark Janes wrote:
>> JUnit has no concept of "warning".  It supports the following
>> statuses:
>>
>>  - skip
>>  - success
>>  - fail
>>  - error
>>
>> dEQP has been found to intermittently emit warnings for passed tests,
>> and this status is accurately represented in piglit json.  However,
>> current JUnit transforms them into failures.
>
> Isn't that something the dEQP runner should take care of? If warnings
> aren't valid there, then just don't flag any test results as warning. At
> least for other cases I'd count gpu hangs in dmesg very much as a failure.
> Hence this result frobbing shouldn't be done here imo.

IMHO there are two separate and unrelated issues which are being
conflated here. The two issues are:

(a) junit conversion of piglit statuses is wrong
(b) deqp tests return random statuses sometimes for no good reason [I
haven't personally verified this, just going by word-of-mouth here.]

Both are issues, and both ought to be addressed.

> -Daniel
>
>>
>> A test which emits a warning is more accurately represented as
>> "success" in JUnit.
>> ---
>>  framework/backends/junit.py | 3 +--
>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>
>> diff --git a/framework/backends/junit.py b/framework/backends/junit.py
>> index 632e516..7499829 100644
>> --- a/framework/backends/junit.py
>> +++ b/framework/backends/junit.py
>> @@ -142,8 +142,7 @@ class JUnitBackend(FileBackend):
>>              if data['result'] == 'skip':
>>                  res = etree.SubElement(element, 'skipped')
>>
>> -            elif data['result'] in ['warn', 'fail', 'dmesg-warn',
>> -                                    'dmesg-fail']:
>> +            elif data['result'] in ['fail', 'dmesg-fail']:
>>                  if expected_result == "failure":
>>                      err.text += "\n\nWARN: passing test as an expected failure"
>>                      res = etree.SubElement(element, 'skipped',
>> --
>> 2.1.4
>>
>> _______________________________________________
>> Piglit mailing list
>> Piglit at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/piglit
>
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/piglit