[Piglit] [PATCH] Interpret dEQP "QualityWarning" as pass

[Mark Janes]

Ilia Mirkin <imirkin at alum.mit.edu> writes:

> On Fri, Jun 26, 2015 at 2:44 PM, Dylan Baker <baker.dylan.c at gmail.com> wrote:
>>> That's fine, but then that's the justification, not "junit interprets
>>> warn as fail". Note that some (regular) piglit tests will also emit
>>> "warn", and they don't mean "fail". I believe that junit needs to be
>>> fixed to not interpret warn as fail irrespective of what happens in
>>> deqp. But for the record, I don't use junit.
>>>
>>> I'm not sure how I feel about the "warn" output in piglit in general,
>>> it seems a bit weird. Bogus and intermittent warnings certainly sound
>>> like a problem, and perhaps that's reason enough to just ignore them.
>>
>> I'm fine with either solution.
>>
>> Warn is kind of an odd status. Basically if a test passes, but there is
>> anything unexpected in stderr, the test will get marked warn. (I think
>> some other suites use it differently, maybe igt?)
>
> Well, with the GL suite, a test can on its own decide to return a
> "warn" status. The framework doesn't automatically add that in (or at
> least didn't before).
>
>   -ilia

And this is why "warn status" is problematic in a test suite.  Consider
the case where a commit introduces a new warning.  Should a bug be
written up?  It depends: as with compiler warnings, some projects may
want to be more strict, and disallow warnings.  Other projects may not
mind warnings, with the result that test writers emit warnings that are
not actionable.

At least with piglit the warnings are stable, so any CI displaying JUnit
trends will not report spurious regressions.  Additionally, the
developers can easily remedy warnings which are intermittent.  For this
reason, warnings-as-errors provides a sharper edge with with to catch
regressions in the core piglit test suite (caveat: I'm not sure how
dmesg-warn is handled for tests executed in parallel).

dEQP has 2 issues that work against using warnings as errors:

 * warnings are intermittent, at least on intel hardware.  Minimal
   investigation by Chad indicated that the warnings are bogus.

 * developers don't have easy access to upstream, to improve the tests.

There may be no optimal, consistent warning policy for both test suites.
I'm inclined to turn down the warning noise emitted by dEQP, because it
is external to piglit and can't be easily fixed.  Even at Google, dEQP
runners blacklist thousands of tests because they are noisy.

-Mark