[Piglit] Plan for enabling more tests in concurrent mode

[Paul Berry]

On 24 August 2012 17:28, Chris Forbes <chrisf at ijw.co.nz> wrote:

> > BTW, we are hoping to reduce the use of glutKeyboardFunc as much as
> possible
> > because it gets in the use of Waffle, so if you happen to notice some
> > opportunities to get rid of glutKeyboardFunc entirely (or at least take
> some
> > steps in that direction), feel free.
>
> There are still a bunch of tests which are actually usefully
> interactive in non-auto mode -- what's the accepted way of dealing
> with those?
> Rendering all the possibilities in a grid in one frame seems vaguely
> reasonable.
>

There's two downsides to rendering all the possibilities in a grid: (a) if
the grid gets too big, the test will start failing on small screens, and
(b) if one of the possibilities fails, the whole test will fail and we
won't get any information about the other possibilities.  (b) is
particularly problematic because sometimes one of the possibilities fails
due to a known bug that is hard to fix or not worth fixing (e.g. sometimes
a certain rare corner case fails due to a hardware bug, and we can't fix it
in the driver without a massive performance penalty).  In those situations,
we still want Piglit to alert us if one of the other possibilities
regresses--if we render all possibilities to a grid and produce a single
pass/fail result, Piglit won't notice when one failure turns into two.

Also, in my experience even the tests that seem to be usefully interactive
via the keyboard are less useful than they at first seem, because when you
are actively pursuing a bug (e.g. when bisecting to find the commit that
caused a failure) you have to fire up the test and then press a sequence of
keys before you can see whether the output changed.  Being able to select
the exact test case you want on the command line is far easier to repeat.
I've had some hallway conversations with Eric and Chad about the tests that
use KeyboardFuncs and I think we're all on the same page about this (please
speak up if I'm misconstruing the consensus).

FYI, a good example to look at is any of the tests that include
fbo-formats.h (for example, fbo-alphatest-formats).  These tests take an
optional category on the command line (if no category is specified, they
test core formats).  When run interactively, they test one format at a
time, and keypresses can be used to switch between formats within a
category, or between categories.  When run with "-auto" (i.e. during a
normal piglit run), they test all the formats in the specified category and
give an aggregate result (leading to problem (b) above).  My personal
preference would be to modify these tests so that the command line
parameter is mandatory, and it specifies the exact format to test rather
than a category.  That would eliminate the need for a KeyboardFunc *and*
solve problem (b).


>
> > Are you talking about adding calls to the piglit_*_projection functions,
> or
> > simply moving the existing calls from the reshape functions to
> > piglit_display so that the reshape functions can be removed?  I'm
> definitely
> > in favor of the latter.  But adding piglit_*_projection function calls is
> > problematic because those functions rely on deprecated functionality, and
> > we're trying as much as possible to reduce Piglit's dependence on
> deprecated
> > functionality so that we can start testing GLES 2.0, GLES 3.0, and
> forward
> > compatible contexts.
>
> Still better to have the deprecated calls in *one* place that can be
> converted to setting uniforms than having
> glMatrixMode/glLoadIdentity/glOrtho sprayed everywhere, though, isn't
> it?
>
> -- Chris
>

Agreed, calling wrapper functions that call deprecated functions is *way*
better than calling the deprecated functions themselves (except in the case
where the deprecated function is the thing we're trying to test, of
course).  As long as you're not planning to add calls to
piglit_*_projection to tests that previously avoided the matrix stack, then
your plan sounds good to me.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/piglit/attachments/20120824/402ac203/attachment.html>