[Piglit] [PATCH] Interpret dEQP "QualityWarning" as pass

Ilia Mirkin imirkin at alum.mit.edu
Fri Jun 26 14:02:46 PDT 2015

On Fri, Jun 26, 2015 at 4:02 PM, Mark Janes <mark.a.janes at intel.com> wrote:
> Ilia Mirkin <imirkin at alum.mit.edu> writes:
>> On Fri, Jun 26, 2015 at 2:44 PM, Dylan Baker <baker.dylan.c at gmail.com> wrote:
>>>> That's fine, but then that's the justification, not "junit interprets
>>>> warn as fail". Note that some (regular) piglit tests will also emit
>>>> "warn", and they don't mean "fail". I believe that junit needs to be
>>>> fixed to not interpret warn as fail irrespective of what happens in
>>>> deqp. But for the record, I don't use junit.
>>>> I'm not sure how I feel about the "warn" output in piglit in general,
>>>> it seems a bit weird. Bogus and intermittent warnings certainly sound
>>>> like a problem, and perhaps that's reason enough to just ignore them.
>>> I'm fine with either solution.
>>> Warn is kind of an odd status. Basically if a test passes, but there is
>>> anything unexpected in stderr, the test will get marked warn. (I think
>>> some other suites use it differently, maybe igt?)
>> Well, with the GL suite, a test can on its own decide to return a
>> "warn" status. The framework doesn't automatically add that in (or at
>> least didn't before).
>>   -ilia
> And this is why "warn status" is problematic in a test suite.  Consider
> the case where a commit introduces a new warning.  Should a bug be
> written up?  It depends: as with compiler warnings, some projects may
> want to be more strict, and disallow warnings.  Other projects may not
> mind warnings, with the result that test writers emit warnings that are
> not actionable.

Defaults are important. The default is "warn is ok". That's how I see
it from piglit. I'd do the split between warn and dmesg-warn. The
latter indicates a real issue, whereas warn indicates a highly
theoretical one. If you add a flag to treat "warn" as error optionally
(-Werror style), I certainly wouldn't object.

> At least with piglit the warnings are stable, so any CI displaying JUnit
> trends will not report spurious regressions.  Additionally, the
> developers can easily remedy warnings which are intermittent.  For this
> reason, warnings-as-errors provides a sharper edge with with to catch
> regressions in the core piglit test suite (caveat: I'm not sure how
> dmesg-warn is handled for tests executed in parallel).
> dEQP has 2 issues that work against using warnings as errors:
>  * warnings are intermittent, at least on intel hardware.  Minimal
>    investigation by Chad indicated that the warnings are bogus.
>  * developers don't have easy access to upstream, to improve the tests.
> There may be no optimal, consistent warning policy for both test suites.
> I'm inclined to turn down the warning noise emitted by dEQP, because it
> is external to piglit and can't be easily fixed.  Even at Google, dEQP
> runners blacklist thousands of tests because they are noisy.
> -Mark

If warns from deqp are useless, then those should also be disabled.
But that has no bearing on what junit does with warns...


More information about the Piglit mailing list