[Piglit] [PATCH] Interpret dEQP "QualityWarning" as pass

[Ilia Mirkin]

On Fri, Jun 26, 2015 at 4:02 PM, Mark Janes <mark.a.janes at intel.com> wrote:
> Ilia Mirkin <imirkin at alum.mit.edu> writes:
>
>> On Fri, Jun 26, 2015 at 2:44 PM, Dylan Baker <baker.dylan.c at gmail.com> wrote:
>>>> That's fine, but then that's the justification, not "junit interprets
>>>> warn as fail". Note that some (regular) piglit tests will also emit
>>>> "warn", and they don't mean "fail". I believe that junit needs to be
>>>> fixed to not interpret warn as fail irrespective of what happens in
>>>> deqp. But for the record, I don't use junit.
>>>>
>>>> I'm not sure how I feel about the "warn" output in piglit in general,
>>>> it seems a bit weird. Bogus and intermittent warnings certainly sound
>>>> like a problem, and perhaps that's reason enough to just ignore them.
>>>
>>> I'm fine with either solution.
>>>
>>> Warn is kind of an odd status. Basically if a test passes, but there is
>>> anything unexpected in stderr, the test will get marked warn. (I think
>>> some other suites use it differently, maybe igt?)
>>
>> Well, with the GL suite, a test can on its own decide to return a
>> "warn" status. The framework doesn't automatically add that in (or at
>> least didn't before).
>>
>>   -ilia
>
> And this is why "warn status" is problematic in a test suite.  Consider
> the case where a commit introduces a new warning.  Should a bug be
> written up?  It depends: as with compiler warnings, some projects may
> want to be more strict, and disallow warnings.  Other projects may not
> mind warnings, with the result that test writers emit warnings that are
> not actionable.

Defaults are important. The default is "warn is ok". That's how I see
it from piglit. I'd do the split between warn and dmesg-warn. The
latter indicates a real issue, whereas warn indicates a highly
theoretical one. If you add a flag to treat "warn" as error optionally
(-Werror style), I certainly wouldn't object.

>
> At least with piglit the warnings are stable, so any CI displaying JUnit
> trends will not report spurious regressions.  Additionally, the
> developers can easily remedy warnings which are intermittent.  For this
> reason, warnings-as-errors provides a sharper edge with with to catch
> regressions in the core piglit test suite (caveat: I'm not sure how
> dmesg-warn is handled for tests executed in parallel).
>
> dEQP has 2 issues that work against using warnings as errors:
>
>  * warnings are intermittent, at least on intel hardware.  Minimal
>    investigation by Chad indicated that the warnings are bogus.
>
>  * developers don't have easy access to upstream, to improve the tests.
>
> There may be no optimal, consistent warning policy for both test suites.
> I'm inclined to turn down the warning noise emitted by dEQP, because it
> is external to piglit and can't be easily fixed.  Even at Google, dEQP
> runners blacklist thousands of tests because they are noisy.
>
> -Mark

If warns from deqp are useless, then those should also be disabled.
But that has no bearing on what junit does with warns...

  -ilia