[Piglit] [PATCH] Interpret dEQP "QualityWarning" as pass

[Ilia Mirkin]

On Fri, Jun 26, 2015 at 12:33 PM, Mark Janes <mark.a.janes at intel.com> wrote:
> Ilia Mirkin <imirkin at alum.mit.edu> writes:
>
>> On Thu, Jun 25, 2015 at 8:52 PM, Mark Janes <mark.a.janes at intel.com> wrote:
>>> Many tests in the functional.fbo.completeness.attachment_combinations
>>> report "QualityWarning" intermittently.  This particular warning is
>>> emitted because dEQP attempted to create an unsupported framebuffer.
>>>
>>> "warning" is interpreted as "failure" in junit, but this output from
>>> dEQP is clearly not a failure.
>>
>> So... fix junit. "warn" is not a failure output by piglit.
>>
>
> I'm fine with changing piglit's junit back end to interpret "warn" as a
> junit "pass".  Changing the dEQP wrapper has smaller scope.  Chad
> recommended changing deqp.py because:
>
>  * dEQP warnings appear to be bogus
>
>  * dEQP warnings are intermittent

That's fine, but then that's the justification, not "junit interprets
warn as fail". Note that some (regular) piglit tests will also emit
"warn", and they don't mean "fail". I believe that junit needs to be
fixed to not interpret warn as fail irrespective of what happens in
deqp. But for the record, I don't use junit.

I'm not sure how I feel about the "warn" output in piglit in general,
it seems a bit weird. Bogus and intermittent warnings certainly sound
like a problem, and perhaps that's reason enough to just ignore them.

>
>  * dEQP has fewer users at this point (is anyone running it?)

I know Rob has run it once or twice on freedreno.

>
> I could work around these warnings by not running the tests which
> produce warnings in Intel's automation, but then we would never know if
> the tests regress.

Of all the possible solutions, that sounds like by far the worst.

>
> -Mark
>
>>> ---
>>>  framework/test/deqp.py | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/framework/test/deqp.py b/framework/test/deqp.py
>>> index 1462ca2..f4bf1cd 100644
>>> --- a/framework/test/deqp.py
>>> +++ b/framework/test/deqp.py
>>> @@ -105,7 +105,7 @@ class DEQPBaseTest(Test):
>>>      __metaclass__ = abc.ABCMeta
>>>      __RESULT_MAP = {"Pass": "pass",
>>>                      "Fail": "fail",
>>> -                    "QualityWarning": "warn",
>>> +                    "QualityWarning": "pass",
>>>                      "InternalError": "fail",
>>>                      "Crash": "crash",
>>>                      "NotSupported": "skip"}
>>> --
>>> 2.1.4
>>>
>>> _______________________________________________
>>> Piglit mailing list
>>> Piglit at lists.freedesktop.org
>>> http://lists.freedesktop.org/mailman/listinfo/piglit