[Piglit] [PATCH 0/1] Add comprehensive tests of builtin functions with uniform input.

[Paul Berry]

Executive summary:
- Comprehensively tests built-in GLSL 1.10 and 1.20 functions.
- Tests are auto-generated at piglit build time.
- Auto-generation code is organized so that in the future, we can
  easily add more built-in functions to be tested, and more contexts
  in which to test them.
- Adds two dependencies to piglit: Python 2.7 and numpy.
- All tests pass on nVidia's proprietary Linux driver, one failure on
  Mesa with Intel i965.

The following patch adds 462 tests to piglit, which comprehensively
test the behavior of GLSL built-in functions on vertex and fragment
shaders.  The test vectors are auto-generated in Python, by using the
numpy library to simulate the behavior of the built-in functions on a
variety of inputs, and then producing shader_runner tests to verify
the expected behavior.  The numpy library includes support for typical
vector and matrix operations such as dot products and outer
products--this allowed complex built-in functions such as refract() to
be simulated using straightforward Python code, so that we can have
high confidence in the correctness of the test vectors.

Since built-in functions tend to be added in each new version of GLSL,
I've incorporated the auto-generation of tests into the Piglit build
process, so that we can easily add tests of more built-ins in the
future.  You can generate the tests using "make gen-tests" from the
root of the piglit tree (after configuring using cmake).

The auto-generation code depends on Python 2.7 (for the argparse
library) and on the numpy library.  Both of these are standard, mature
packages (Python 2.7 was released a year ago, and numpy has been
around since 1995).  So I'm hoping it won't be a problem adding these
as dependencies, especially since Piglit is used principally by
developers testing cutting-edge drivers.  We could, in principle,
relax the Python 2.7 requirement to 2.6 by using optparse instead of
argparse, but I'm hoping not to have to do so because optparse is
deprecated as of Python 2.7.  I've updated the README to reflect these
dependencies, and added code to CMakeLists.txt to check for them at
configure time.

The set of built-ins currently tested covers all of GLSL 1.10 and GLSL
1.20, with the exception of ftransform(), texture lookup functions,
local differencing functions (dFdx(), dFdy(), and fwidth()), and noise
functions.  Some of these functions were already fairly well-tested in
Piglit; others were lightly tested if at all.  Sometime in the next
few months I plan to expand these tests to cover GLSL 1.30.

Since GL implementations may contain different code paths to support
each built-in function in different contexts, I've separated the test
generation into two files: a back-end, builtin_function.py, which
generates the test vectors themselves, and a front-end,
gen_builtin_uniform_tests.py, which creates shader_runner tests to
test the built-ins on uniform inputs to vertex and fragment shaders.
The principal export of builtin_function.py is a single dictionary,
test_suite, which contains a complete set of test vectors for all
built-in functions.  This should allow us to add front-ends for
testing the built-in functions in different contexts, without having
to revisit the trickier back-end code.  In the next week or two I plan
to add a second front-end to test the behavior of built-in functions
on constant values, as part of some planned work on Mesa's
constant-folding code.

I've validated these tests using nVidia's proprietary Linux driver and
using Mesa with an Intel i965 chipset.  The nVidia driver passes all
tests.  Mesa passes all tests but one, which I'll be submitting a fix
for shortly.

I realize this is a lot of tricky Python code to dump into an
otherwise straightforward project.  Ordinarily I would just check in
the generated tests for easier review.  But in this case, I think that
we are going to get enough advantage out of extending these tests in
the future (both by adding more built-ins and by adding more
front-ends to test them in different contexts) that it's worth
autogenerating the tests at Piglit compile time.

As an aid in review, the complete list of functions tested may be
found by looking at lines beginning in "f(" in
generated_tests/builtin_function.py.  Here is an example:

    f('mod', 2, '1.10', lambda x, y: x-y*np.floor(x/y), [1], [np.linspace(-1.9, 1.9, 4), np.linspace(-2.0, 2.0, 4)])

The meanings of f()'s arguments are explained in its definition; there
are a few variants of f() for testing different types of built-ins.
For this example, the arguments represent:

- The name of the function
- The number of arguments it takes
- The version of GLSL in which it was introduced
- The function's Python equivalent (this is typically either a
  built-in function in numpy or a straightforward transliteration of a
  formula in the GLSL spec).
- If the function has an equivalent form in which some arguments are
  vectors and some arguments are scalars, a list of the indices of
  which arguments are scalars.  So in the case of "mod", this means
  that in addition to mod(genType x, genType y), there is a form
  mod(genType x, float y).
- A list of possible input values to test.  np.linspace(x, y, n) is a
  numpy function that generates a list of n linearly spaced values
  from x to y.  So in the case of mod, the test vectors will exercise
  the cartesian product of 4 possible x values from -1.9 to 1.9, and 4
  possible y values from -2.0 to 2.0.

One of the areas in which I'm most interested in receiving review is
in whether I have chosen an appropiate set of input values for testing
each function.

Here is an example of a test created by gen_builtin_uniform_tests.py.
This example is vs-smoothstep-float-float-vec4.shader_test.  Note that
the filename contains the types of the arguments, so this test
exercises the function smoothstep(float, float, vec4).  I've tried to
make the generated test as straightforward as possible, so that when
failures occur it will be easy to understand what has failed.

[require]
GLSL >= 1.10

[vertex shader]
varying vec4 color;
uniform float arg0;
uniform float arg1;
uniform vec4 arg2;

void main()
{
  gl_Position = gl_Vertex;
  vec4 result = smoothstep(arg0, arg1, arg2);
  result -= -0.5;
  result *= 0.5;
  color = vec4(result);
}

[fragment shader]
varying vec4 color;

void main()
{
  gl_FragColor = color;
}

[test]
uniform float arg0 -1.9
uniform float arg1 -0.633333333333
uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
draw rect -1 -1 2 2
probe rgba 0 0 0.25 0.748979443068 0.75 0.75
uniform float arg0 -1.9
uniform float arg1 0.633333333333
uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
draw rect -1 -1 2 2
probe rgba 1 0 0.25 0.490133856976 0.75 0.75
uniform float arg0 -1.9
uniform float arg1 1.9
uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
draw rect -1 -1 2 2
probe rgba 2 0 0.25 0.373820824761 0.626179175239 0.75
uniform float arg0 -0.633333333333
uniform float arg1 0.633333333333
uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
draw rect -1 -1 2 2
probe rgba 3 0 0.25 0.25 0.75 0.75
uniform float arg0 -0.633333333333
uniform float arg1 1.9
uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
draw rect -1 -1 2 2
probe rgba 4 0 0.25 0.25 0.509866143024 0.75
uniform float arg0 0.633333333333
uniform float arg1 1.9
uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
draw rect -1 -1 2 2
probe rgba 5 0 0.25 0.25 0.251020556932 0.75