[Piglit] [PATCH 0/1] Add comprehensive tests of builtin functions with uniform input.

[Ian Romanick]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/22/2011 02:00 PM, Paul Berry wrote:
> Executive summary:
> - Comprehensively tests built-in GLSL 1.10 and 1.20 functions.
> - Tests are auto-generated at piglit build time.
> - Auto-generation code is organized so that in the future, we can
>   easily add more built-in functions to be tested, and more contexts
>   in which to test them.
> - Adds two dependencies to piglit: Python 2.7 and numpy.
> - All tests pass on nVidia's proprietary Linux driver, one failure on
>   Mesa with Intel i965.

Wow.  This is pretty cool.

I'm sure I'll have more comments next week, but there was one thing
below that jumped out at me...

> The following patch adds 462 tests to piglit, which comprehensively
> test the behavior of GLSL built-in functions on vertex and fragment
> shaders.  The test vectors are auto-generated in Python, by using the
> numpy library to simulate the behavior of the built-in functions on a
> variety of inputs, and then producing shader_runner tests to verify
> the expected behavior.  The numpy library includes support for typical
> vector and matrix operations such as dot products and outer
> products--this allowed complex built-in functions such as refract() to
> be simulated using straightforward Python code, so that we can have
> high confidence in the correctness of the test vectors.
> 
> Since built-in functions tend to be added in each new version of GLSL,
> I've incorporated the auto-generation of tests into the Piglit build
> process, so that we can easily add tests of more built-ins in the
> future.  You can generate the tests using "make gen-tests" from the
> root of the piglit tree (after configuring using cmake).
> 
> The auto-generation code depends on Python 2.7 (for the argparse
> library) and on the numpy library.  Both of these are standard, mature
> packages (Python 2.7 was released a year ago, and numpy has been
> around since 1995).  So I'm hoping it won't be a problem adding these
> as dependencies, especially since Piglit is used principally by
> developers testing cutting-edge drivers.  We could, in principle,
> relax the Python 2.7 requirement to 2.6 by using optparse instead of
> argparse, but I'm hoping not to have to do so because optparse is
> deprecated as of Python 2.7.  I've updated the README to reflect these
> dependencies, and added code to CMakeLists.txt to check for them at
> configure time.
> 
> The set of built-ins currently tested covers all of GLSL 1.10 and GLSL
> 1.20, with the exception of ftransform(), texture lookup functions,
> local differencing functions (dFdx(), dFdy(), and fwidth()), and noise
> functions.  Some of these functions were already fairly well-tested in
> Piglit; others were lightly tested if at all.  Sometime in the next
> few months I plan to expand these tests to cover GLSL 1.30.
> 
> Since GL implementations may contain different code paths to support
> each built-in function in different contexts, I've separated the test
> generation into two files: a back-end, builtin_function.py, which
> generates the test vectors themselves, and a front-end,
> gen_builtin_uniform_tests.py, which creates shader_runner tests to
> test the built-ins on uniform inputs to vertex and fragment shaders.
> The principal export of builtin_function.py is a single dictionary,
> test_suite, which contains a complete set of test vectors for all
> built-in functions.  This should allow us to add front-ends for
> testing the built-in functions in different contexts, without having
> to revisit the trickier back-end code.  In the next week or two I plan
> to add a second front-end to test the behavior of built-in functions
> on constant values, as part of some planned work on Mesa's
> constant-folding code.
> 
> I've validated these tests using nVidia's proprietary Linux driver and
> using Mesa with an Intel i965 chipset.  The nVidia driver passes all
> tests.  Mesa passes all tests but one, which I'll be submitting a fix
> for shortly.
> 
> I realize this is a lot of tricky Python code to dump into an
> otherwise straightforward project.  Ordinarily I would just check in
> the generated tests for easier review.  But in this case, I think that
> we are going to get enough advantage out of extending these tests in
> the future (both by adding more built-ins and by adding more
> front-ends to test them in different contexts) that it's worth
> autogenerating the tests at Piglit compile time.
> 
> As an aid in review, the complete list of functions tested may be
> found by looking at lines beginning in "f(" in
> generated_tests/builtin_function.py.  Here is an example:
> 
>     f('mod', 2, '1.10', lambda x, y: x-y*np.floor(x/y), [1], [np.linspace(-1.9, 1.9, 4), np.linspace(-2.0, 2.0, 4)])
> 
> The meanings of f()'s arguments are explained in its definition; there
> are a few variants of f() for testing different types of built-ins.
> For this example, the arguments represent:
> 
> - The name of the function
> - The number of arguments it takes
> - The version of GLSL in which it was introduced
> - The function's Python equivalent (this is typically either a
>   built-in function in numpy or a straightforward transliteration of a
>   formula in the GLSL spec).
> - If the function has an equivalent form in which some arguments are
>   vectors and some arguments are scalars, a list of the indices of
>   which arguments are scalars.  So in the case of "mod", this means
>   that in addition to mod(genType x, genType y), there is a form
>   mod(genType x, float y).
> - A list of possible input values to test.  np.linspace(x, y, n) is a
>   numpy function that generates a list of n linearly spaced values
>   from x to y.  So in the case of mod, the test vectors will exercise
>   the cartesian product of 4 possible x values from -1.9 to 1.9, and 4
>   possible y values from -2.0 to 2.0.
> 
> One of the areas in which I'm most interested in receiving review is
> in whether I have chosen an appropiate set of input values for testing
> each function.
> 
> Here is an example of a test created by gen_builtin_uniform_tests.py.
> This example is vs-smoothstep-float-float-vec4.shader_test.  Note that
> the filename contains the types of the arguments, so this test
> exercises the function smoothstep(float, float, vec4).  I've tried to
> make the generated test as straightforward as possible, so that when
> failures occur it will be easy to understand what has failed.
> 
> [require]
> GLSL >= 1.10
> 
> [vertex shader]
> varying vec4 color;
> uniform float arg0;
> uniform float arg1;
> uniform vec4 arg2;
> 
> void main()
> {
>   gl_Position = gl_Vertex;
>   vec4 result = smoothstep(arg0, arg1, arg2);
>   result -= -0.5;
>   result *= 0.5;
>   color = vec4(result);
> }
> 
> [fragment shader]
> varying vec4 color;
> 
> void main()
> {
>   gl_FragColor = color;
> }
> 
> [test]
> uniform float arg0 -1.9
> uniform float arg1 -0.633333333333
> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
> draw rect -1 -1 2 2
> probe rgba 0 0 0.25 0.748979443068 0.75 0.75

Checks like this are a big problem for shaders.  With 8-bit color
outputs, checks that require any precision are doomed.  As you noticed,
you can only validate answers on the range [0,1].  Moreover, you can
only verify that the calculated answer is within 1/255 of correct.
That's just not good enough for most things.

Most of the time you want to check that a calculation is correct within
a certain tolerance.  In those cases, you should do exactly that in the
shader.  Some of the non-constant indexing tests that I wrote do this.
In one of the cases, vs-varying-mat4-wr.shader_test, I want to check
that a matrix passed to fragment shader has a varying had correct
values.  I checked that by doing a calculation with that matrix and
verifying the correctness of that result:

void main()
{
    gl_FragColor = (abs(distance(dst_matrix * v, expect)) < 1e-6)
        ? vec4(0.0, 1.0, 0.0, 1.0) : vec4(1.0, 0.0, 0.0, 1.0);
}

Of course, this assumes that abs() and distance() work correctly. :)

> uniform float arg0 -1.9
> uniform float arg1 0.633333333333
> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
> draw rect -1 -1 2 2
> probe rgba 1 0 0.25 0.490133856976 0.75 0.75
> uniform float arg0 -1.9
> uniform float arg1 1.9
> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
> draw rect -1 -1 2 2
> probe rgba 2 0 0.25 0.373820824761 0.626179175239 0.75
> uniform float arg0 -0.633333333333
> uniform float arg1 0.633333333333
> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
> draw rect -1 -1 2 2
> probe rgba 3 0 0.25 0.25 0.75 0.75
> uniform float arg0 -0.633333333333
> uniform float arg1 1.9
> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
> draw rect -1 -1 2 2
> probe rgba 4 0 0.25 0.25 0.509866143024 0.75
> uniform float arg0 0.633333333333
> uniform float arg1 1.9
> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
> draw rect -1 -1 2 2
> probe rgba 5 0 0.25 0.25 0.251020556932 0.75
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAk4qijIACgkQX1gOwKyEAw+fyQCghwXcwTjjaLmgcEkQrUdBjvxc
2RgAnRdRXidJmvDrsz7DUI0WaisFk8tP
=ojD2
-----END PGP SIGNATURE-----