[Piglit] [PATCH 0/1] Add comprehensive tests of builtin functions with uniform input.

[Ian Romanick]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 08/05/2011 01:09 PM, Paul Berry wrote:
> On 22 July 2011 14:00, Paul Berry <stereotype441 at gmail.com> wrote:
>> Executive summary:
>> - Comprehensively tests built-in GLSL 1.10 and 1.20 functions.
>> - Tests are auto-generated at piglit build time.
>> - Auto-generation code is organized so that in the future, we can
>>  easily add more built-in functions to be tested, and more contexts
>>  in which to test them.
>> - Adds two dependencies to piglit: Python 2.7 and numpy.
>> - All tests pass on nVidia's proprietary Linux driver, one failure on
>>  Mesa with Intel i965.
>>
>> The following patch adds 462 tests to piglit, which comprehensively
>> test the behavior of GLSL built-in functions on vertex and fragment
>> shaders.  The test vectors are auto-generated in Python, by using the
>> numpy library to simulate the behavior of the built-in functions on a
>> variety of inputs, and then producing shader_runner tests to verify
>> the expected behavior.  The numpy library includes support for typical
>> vector and matrix operations such as dot products and outer
>> products--this allowed complex built-in functions such as refract() to
>> be simulated using straightforward Python code, so that we can have
>> high confidence in the correctness of the test vectors.
>>
>> Since built-in functions tend to be added in each new version of GLSL,
>> I've incorporated the auto-generation of tests into the Piglit build
>> process, so that we can easily add tests of more built-ins in the
>> future.  You can generate the tests using "make gen-tests" from the
>> root of the piglit tree (after configuring using cmake).
>>
>> The auto-generation code depends on Python 2.7 (for the argparse
>> library) and on the numpy library.  Both of these are standard, mature
>> packages (Python 2.7 was released a year ago, and numpy has been
>> around since 1995).  So I'm hoping it won't be a problem adding these
>> as dependencies, especially since Piglit is used principally by
>> developers testing cutting-edge drivers.  We could, in principle,
>> relax the Python 2.7 requirement to 2.6 by using optparse instead of
>> argparse, but I'm hoping not to have to do so because optparse is
>> deprecated as of Python 2.7.  I've updated the README to reflect these
>> dependencies, and added code to CMakeLists.txt to check for them at
>> configure time.
>>
>> The set of built-ins currently tested covers all of GLSL 1.10 and GLSL
>> 1.20, with the exception of ftransform(), texture lookup functions,
>> local differencing functions (dFdx(), dFdy(), and fwidth()), and noise
>> functions.  Some of these functions were already fairly well-tested in
>> Piglit; others were lightly tested if at all.  Sometime in the next
>> few months I plan to expand these tests to cover GLSL 1.30.
>>
>> Since GL implementations may contain different code paths to support
>> each built-in function in different contexts, I've separated the test
>> generation into two files: a back-end, builtin_function.py, which
>> generates the test vectors themselves, and a front-end,
>> gen_builtin_uniform_tests.py, which creates shader_runner tests to
>> test the built-ins on uniform inputs to vertex and fragment shaders.
>> The principal export of builtin_function.py is a single dictionary,
>> test_suite, which contains a complete set of test vectors for all
>> built-in functions.  This should allow us to add front-ends for
>> testing the built-in functions in different contexts, without having
>> to revisit the trickier back-end code.  In the next week or two I plan
>> to add a second front-end to test the behavior of built-in functions
>> on constant values, as part of some planned work on Mesa's
>> constant-folding code.
>>
>> I've validated these tests using nVidia's proprietary Linux driver and
>> using Mesa with an Intel i965 chipset.  The nVidia driver passes all
>> tests.  Mesa passes all tests but one, which I'll be submitting a fix
>> for shortly.
>>
>> I realize this is a lot of tricky Python code to dump into an
>> otherwise straightforward project.  Ordinarily I would just check in
>> the generated tests for easier review.  But in this case, I think that
>> we are going to get enough advantage out of extending these tests in
>> the future (both by adding more built-ins and by adding more
>> front-ends to test them in different contexts) that it's worth
>> autogenerating the tests at Piglit compile time.
>>
>> As an aid in review, the complete list of functions tested may be
>> found by looking at lines beginning in "f(" in
>> generated_tests/builtin_function.py.  Here is an example:
>>
>>    f('mod', 2, '1.10', lambda x, y: x-y*np.floor(x/y), [1], [np.linspace(-1.9, 1.9, 4), np.linspace(-2.0, 2.0, 4)])
>>
>> The meanings of f()'s arguments are explained in its definition; there
>> are a few variants of f() for testing different types of built-ins.
>> For this example, the arguments represent:
>>
>> - The name of the function
>> - The number of arguments it takes
>> - The version of GLSL in which it was introduced
>> - The function's Python equivalent (this is typically either a
>>  built-in function in numpy or a straightforward transliteration of a
>>  formula in the GLSL spec).
>> - If the function has an equivalent form in which some arguments are
>>  vectors and some arguments are scalars, a list of the indices of
>>  which arguments are scalars.  So in the case of "mod", this means
>>  that in addition to mod(genType x, genType y), there is a form
>>  mod(genType x, float y).
>> - A list of possible input values to test.  np.linspace(x, y, n) is a
>>  numpy function that generates a list of n linearly spaced values
>>  from x to y.  So in the case of mod, the test vectors will exercise
>>  the cartesian product of 4 possible x values from -1.9 to 1.9, and 4
>>  possible y values from -2.0 to 2.0.
>>
>> One of the areas in which I'm most interested in receiving review is
>> in whether I have chosen an appropiate set of input values for testing
>> each function.
>>
>> Here is an example of a test created by gen_builtin_uniform_tests.py.
>> This example is vs-smoothstep-float-float-vec4.shader_test.  Note that
>> the filename contains the types of the arguments, so this test
>> exercises the function smoothstep(float, float, vec4).  I've tried to
>> make the generated test as straightforward as possible, so that when
>> failures occur it will be easy to understand what has failed.
>>
>> [require]
>> GLSL >= 1.10
>>
>> [vertex shader]
>> varying vec4 color;
>> uniform float arg0;
>> uniform float arg1;
>> uniform vec4 arg2;
>>
>> void main()
>> {
>>  gl_Position = gl_Vertex;
>>  vec4 result = smoothstep(arg0, arg1, arg2);
>>  result -= -0.5;
>>  result *= 0.5;
>>  color = vec4(result);
>> }
>>
>> [fragment shader]
>> varying vec4 color;
>>
>> void main()
>> {
>>  gl_FragColor = color;
>> }
>>
>> [test]
>> uniform float arg0 -1.9
>> uniform float arg1 -0.633333333333
>> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
>> draw rect -1 -1 2 2
>> probe rgba 0 0 0.25 0.748979443068 0.75 0.75
>> uniform float arg0 -1.9
>> uniform float arg1 0.633333333333
>> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
>> draw rect -1 -1 2 2
>> probe rgba 1 0 0.25 0.490133856976 0.75 0.75
>> uniform float arg0 -1.9
>> uniform float arg1 1.9
>> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
>> draw rect -1 -1 2 2
>> probe rgba 2 0 0.25 0.373820824761 0.626179175239 0.75
>> uniform float arg0 -0.633333333333
>> uniform float arg1 0.633333333333
>> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
>> draw rect -1 -1 2 2
>> probe rgba 3 0 0.25 0.25 0.75 0.75
>> uniform float arg0 -0.633333333333
>> uniform float arg1 1.9
>> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
>> draw rect -1 -1 2 2
>> probe rgba 4 0 0.25 0.25 0.509866143024 0.75
>> uniform float arg0 0.633333333333
>> uniform float arg1 1.9
>> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
>> draw rect -1 -1 2 2
>> probe rgba 5 0 0.25 0.25 0.251020556932 0.75
>>
> 
> Ok, these tests (as modified according to people's suggestions) have
> been pushed.  Let me know if you run into any trouble with them and
> I'll be glad to help.

I was trying to run these on AMD's closed-source driver, but I was
running into other piglit troubles.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAk48Tm0ACgkQX1gOwKyEAw8n6ACcCEpOm06y9d9IFjSQMpmasYJg
UnkAn3n6XxZbjrLdm55gJN6Vs04vMTwd
=UMhu
-----END PGP SIGNATURE-----