[Piglit] [PATCH 0/1] Add comprehensive tests of builtin functions with uniform input.

[Paul Berry]

On 22 July 2011 14:00, Paul Berry <stereotype441 at gmail.com> wrote:
> Executive summary:
> - Comprehensively tests built-in GLSL 1.10 and 1.20 functions.
> - Tests are auto-generated at piglit build time.
> - Auto-generation code is organized so that in the future, we can
>  easily add more built-in functions to be tested, and more contexts
>  in which to test them.
> - Adds two dependencies to piglit: Python 2.7 and numpy.
> - All tests pass on nVidia's proprietary Linux driver, one failure on
>  Mesa with Intel i965.
>
> The following patch adds 462 tests to piglit, which comprehensively
> test the behavior of GLSL built-in functions on vertex and fragment
> shaders.  The test vectors are auto-generated in Python, by using the
> numpy library to simulate the behavior of the built-in functions on a
> variety of inputs, and then producing shader_runner tests to verify
> the expected behavior.  The numpy library includes support for typical
> vector and matrix operations such as dot products and outer
> products--this allowed complex built-in functions such as refract() to
> be simulated using straightforward Python code, so that we can have
> high confidence in the correctness of the test vectors.
>
> Since built-in functions tend to be added in each new version of GLSL,
> I've incorporated the auto-generation of tests into the Piglit build
> process, so that we can easily add tests of more built-ins in the
> future.  You can generate the tests using "make gen-tests" from the
> root of the piglit tree (after configuring using cmake).
>
> The auto-generation code depends on Python 2.7 (for the argparse
> library) and on the numpy library.  Both of these are standard, mature
> packages (Python 2.7 was released a year ago, and numpy has been
> around since 1995).  So I'm hoping it won't be a problem adding these
> as dependencies, especially since Piglit is used principally by
> developers testing cutting-edge drivers.  We could, in principle,
> relax the Python 2.7 requirement to 2.6 by using optparse instead of
> argparse, but I'm hoping not to have to do so because optparse is
> deprecated as of Python 2.7.  I've updated the README to reflect these
> dependencies, and added code to CMakeLists.txt to check for them at
> configure time.
>
> The set of built-ins currently tested covers all of GLSL 1.10 and GLSL
> 1.20, with the exception of ftransform(), texture lookup functions,
> local differencing functions (dFdx(), dFdy(), and fwidth()), and noise
> functions.  Some of these functions were already fairly well-tested in
> Piglit; others were lightly tested if at all.  Sometime in the next
> few months I plan to expand these tests to cover GLSL 1.30.
>
> Since GL implementations may contain different code paths to support
> each built-in function in different contexts, I've separated the test
> generation into two files: a back-end, builtin_function.py, which
> generates the test vectors themselves, and a front-end,
> gen_builtin_uniform_tests.py, which creates shader_runner tests to
> test the built-ins on uniform inputs to vertex and fragment shaders.
> The principal export of builtin_function.py is a single dictionary,
> test_suite, which contains a complete set of test vectors for all
> built-in functions.  This should allow us to add front-ends for
> testing the built-in functions in different contexts, without having
> to revisit the trickier back-end code.  In the next week or two I plan
> to add a second front-end to test the behavior of built-in functions
> on constant values, as part of some planned work on Mesa's
> constant-folding code.
>
> I've validated these tests using nVidia's proprietary Linux driver and
> using Mesa with an Intel i965 chipset.  The nVidia driver passes all
> tests.  Mesa passes all tests but one, which I'll be submitting a fix
> for shortly.
>
> I realize this is a lot of tricky Python code to dump into an
> otherwise straightforward project.  Ordinarily I would just check in
> the generated tests for easier review.  But in this case, I think that
> we are going to get enough advantage out of extending these tests in
> the future (both by adding more built-ins and by adding more
> front-ends to test them in different contexts) that it's worth
> autogenerating the tests at Piglit compile time.
>
> As an aid in review, the complete list of functions tested may be
> found by looking at lines beginning in "f(" in
> generated_tests/builtin_function.py.  Here is an example:
>
>    f('mod', 2, '1.10', lambda x, y: x-y*np.floor(x/y), [1], [np.linspace(-1.9, 1.9, 4), np.linspace(-2.0, 2.0, 4)])
>
> The meanings of f()'s arguments are explained in its definition; there
> are a few variants of f() for testing different types of built-ins.
> For this example, the arguments represent:
>
> - The name of the function
> - The number of arguments it takes
> - The version of GLSL in which it was introduced
> - The function's Python equivalent (this is typically either a
>  built-in function in numpy or a straightforward transliteration of a
>  formula in the GLSL spec).
> - If the function has an equivalent form in which some arguments are
>  vectors and some arguments are scalars, a list of the indices of
>  which arguments are scalars.  So in the case of "mod", this means
>  that in addition to mod(genType x, genType y), there is a form
>  mod(genType x, float y).
> - A list of possible input values to test.  np.linspace(x, y, n) is a
>  numpy function that generates a list of n linearly spaced values
>  from x to y.  So in the case of mod, the test vectors will exercise
>  the cartesian product of 4 possible x values from -1.9 to 1.9, and 4
>  possible y values from -2.0 to 2.0.
>
> One of the areas in which I'm most interested in receiving review is
> in whether I have chosen an appropiate set of input values for testing
> each function.
>
> Here is an example of a test created by gen_builtin_uniform_tests.py.
> This example is vs-smoothstep-float-float-vec4.shader_test.  Note that
> the filename contains the types of the arguments, so this test
> exercises the function smoothstep(float, float, vec4).  I've tried to
> make the generated test as straightforward as possible, so that when
> failures occur it will be easy to understand what has failed.
>
> [require]
> GLSL >= 1.10
>
> [vertex shader]
> varying vec4 color;
> uniform float arg0;
> uniform float arg1;
> uniform vec4 arg2;
>
> void main()
> {
>  gl_Position = gl_Vertex;
>  vec4 result = smoothstep(arg0, arg1, arg2);
>  result -= -0.5;
>  result *= 0.5;
>  color = vec4(result);
> }
>
> [fragment shader]
> varying vec4 color;
>
> void main()
> {
>  gl_FragColor = color;
> }
>
> [test]
> uniform float arg0 -1.9
> uniform float arg1 -0.633333333333
> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
> draw rect -1 -1 2 2
> probe rgba 0 0 0.25 0.748979443068 0.75 0.75
> uniform float arg0 -1.9
> uniform float arg1 0.633333333333
> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
> draw rect -1 -1 2 2
> probe rgba 1 0 0.25 0.490133856976 0.75 0.75
> uniform float arg0 -1.9
> uniform float arg1 1.9
> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
> draw rect -1 -1 2 2
> probe rgba 2 0 0.25 0.373820824761 0.626179175239 0.75
> uniform float arg0 -0.633333333333
> uniform float arg1 0.633333333333
> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
> draw rect -1 -1 2 2
> probe rgba 3 0 0.25 0.25 0.75 0.75
> uniform float arg0 -0.633333333333
> uniform float arg1 1.9
> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
> draw rect -1 -1 2 2
> probe rgba 4 0 0.25 0.25 0.509866143024 0.75
> uniform float arg0 0.633333333333
> uniform float arg1 1.9
> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
> draw rect -1 -1 2 2
> probe rgba 5 0 0.25 0.25 0.251020556932 0.75
>

Ok, these tests (as modified according to people's suggestions) have
been pushed.  Let me know if you run into any trouble with them and
I'll be glad to help.

Paul