[Piglit] [PATCH] arb_shader_precision: add tests for floating point precision
Micah Fedke
micah.fedke at collabora.co.uk
Fri Dec 19 09:01:05 PST 2014
This generated_tests script creates a suite of tests that measure the
floating point precision of most GLSL built-ins, according to
ARB_shader_precision. Test vectors come from builtin_function.py, but
are filtered down to avoid non-float types.
These tests are reporting precision errors in ceil, cross, degrees,
distance, inverse, mix, mod, normalize, op-assign-div, op-assign-mult,
op-div, op-mult, reflect, refract and smoothstep on Ivybridge.
---
Updates since previous (RFC v5):
RFC ended, now submitting as a patch
Suppressed the use of repr() in the generator
- python's repr() does not output enough digits to uniquely
represent a 32 bit float value - the {0:1.8e} format is
used instead
Eliminated the use of distance() for ulps calculations
- this was producing an imprecise result in some situations
Updated the list of failing functions in the description
Note: I am new to the project and don't have commit access.
generated_tests/CMakeLists.txt | 10 +-
generated_tests/gen_shader_precision_tests.py | 159 +++++++++++++++++++++
generated_tests/shader_precision_templates/fs.mako | 127 ++++++++++++++++
generated_tests/shader_precision_templates/gs.mako | 140 ++++++++++++++++++
generated_tests/shader_precision_templates/vs.mako | 136 ++++++++++++++++++
5 files changed, 571 insertions(+), 1 deletion(-)
create mode 100644 generated_tests/gen_shader_precision_tests.py
create mode 100644 generated_tests/shader_precision_templates/fs.mako
create mode 100644 generated_tests/shader_precision_templates/gs.mako
create mode 100644 generated_tests/shader_precision_templates/vs.mako
diff --git a/generated_tests/CMakeLists.txt b/generated_tests/CMakeLists.txt
index 6d27b3e..77ee06c 100644
--- a/generated_tests/CMakeLists.txt
+++ b/generated_tests/CMakeLists.txt
@@ -91,6 +91,13 @@ piglit_make_generated_tests(
constant_array_size_tests_fp64.list
gen_constant_array_size_tests_fp64.py
builtin_function_fp64.py)
+piglit_make_generated_tests(
+ shader_precision_tests.list
+ gen_shader_precision_tests.py
+ builtin_function.py
+ shader_precision_templates/vs.mako
+ shader_precision_templates/fs.mako
+ shader_precision_templates/gs.mako)
# Add a "gen-tests" target that can be used to generate all the
# tests without doing any other compilation.
@@ -113,4 +120,5 @@ add_custom_target(gen-tests ALL
uniform-initializer_tests.list
interpolation-qualifier-built-in-variable.list
builtin_uniform_tests_fp64.list
- constant_array_size_tests_fp64.list)
+ constant_array_size_tests_fp64.list
+ shader_precision_tests.list)
diff --git a/generated_tests/gen_shader_precision_tests.py b/generated_tests/gen_shader_precision_tests.py
new file mode 100644
index 0000000..2eab2b0
--- /dev/null
+++ b/generated_tests/gen_shader_precision_tests.py
@@ -0,0 +1,159 @@
+# coding=utf-8
+#
+# Copyright © 2014 Intel Corporation
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice (including the next
+# paragraph) shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+# DEALINGS IN THE SOFTWARE.
+
+
+"""Generate a set of shader_runner tests for overloaded versions of every
+ built-in function specified in arb_shader_precision, based on the test
+ vectors computed by builtin_function.py, and structured according to the mako
+ templates in shader_precision_templates/.
+
+ The vertex, geometry, and fragment shader types are exercised by each test.
+ In all cases, the inputs to the built-in functions come from uniforms, so
+ that the effectiveness of the test won't be circumvented by constant folding
+ in the GLSL compiler.
+
+ The tests operate by invoking the built-in function in the appropriate
+ shader, calculating any deviance from the expected value (in ulps), comparing
+ the deviance to a supplied tolerance (according to those specified in
+ arb_shader_precision), and then outputting the pass/fail result as a solid
+ rgba color which is then checked using shader_runner's "probe rgba" command.
+
+ For built-in functions whose result type is multi-valued (vec or mat), the
+ tests calculate the error in ulps for each element separately, but compare
+ only the largest error value to the tolerance. This accounts for cases where
+ error varies among the elements of a vec or mat result.
+
+ This program outputs, to stdout, the name of each file it generates.
+"""
+
+from builtin_function import *
+import mako.template
+import os
+import os.path
+
+tolerances = {'pow': 16.0,
+ 'exp': 3.0,
+ 'exp2': 3.0,
+ 'log': 3.0,
+ 'log2': 3.0,
+ 'sqrt': 3.0,
+ 'inversesqrt': 2.0}
+
+trig_builtins = ('sin', 'cos', 'tan',
+ 'asin', 'acos', 'atan',
+ 'sinh', 'cosh', 'tanh',
+ 'asinh', 'acosh', 'atanh')
+
+def make_indexers(signature):
+ """Build a list of strings which index into every possible
+ value of the result. For example, if the result is a vec2,
+ then build the indexers ['[0]', '[1]'].
+ """
+ if signature.rettype.num_cols == 1:
+ col_indexers = ['']
+ else:
+ col_indexers = ['[{0}]'.format(i) for i in xrange(signature.rettype.num_cols)]
+ if signature.rettype.num_rows == 1:
+ row_indexers = ['']
+ else:
+ row_indexers = ['[{0}]'.format(i) for i in xrange(signature.rettype.num_rows)]
+ return [col_indexer + row_indexer
+ for col_indexer in col_indexers for row_indexer in row_indexers]
+
+def shader_runner_type(glsl_type):
+ """Return the appropriate type name necessary for binding a
+ uniform of the given type using shader_runner's "uniform" command.
+ Boolean values and vectors are converted to ints, and square
+ matrices are written in "matNxN" form.
+ """
+ if glsl_type.base_type == glsl_bool:
+ if glsl_type.is_scalar:
+ return 'int'
+ else:
+ return 'ivec{0}'.format(glsl_type.num_rows)
+ elif glsl_type.is_matrix:
+ return 'mat{0}x{1}'.format(glsl_type.num_cols, glsl_type.num_rows)
+ else:
+ return str(glsl_type)
+
+def shader_runner_format(values):
+ """Format the given values for use in a shader_runner "uniform" or
+ "probe rgba" command. Bools are converted to 0's and 1's, and
+ values are separated by spaces.
+ """
+ transformed_values = []
+ retval = ''
+ for value in values:
+ if isinstance(value, (bool, np.bool_)):
+ transformed_values.append(int(value))
+ else:
+ transformed_values.append(value)
+ for x in transformed_values:
+ if isinstance(x,np.float32):
+ retval+=' {0}'.format('{0:1.8e}'.format(x))
+ else:
+ retval+=' {0}'.format(repr(x))
+ return retval
+
+def main():
+ """ Main function """
+
+ for signature, test_vectors in sorted(test_suite.iteritems()):
+ arg_float_check = tuple(
+ arg.base_type == glsl_float for arg in signature.argtypes)
+ # Filter the test vectors down to only those which deal exclusively in float types
+ #and are not trig functions or determinant()
+ indexers = make_indexers(signature)
+ num_elements = signature.rettype.num_cols*signature.rettype.num_rows
+ invocation = signature.template.format( *['arg{0}'.format(i)
+ for i in xrange(len(signature.argtypes))])
+ if (signature.rettype.base_type == glsl_float and
+ arg_float_check and
+ all(arg_float_check) and
+ signature.name not in trig_builtins and
+ signature.name != 'determinant'):
+ for shader_stage in ('vs', 'fs', 'gs'):
+ input_filename = 'shader_precision_templates/{0}.mako'.format(shader_stage)
+ template = mako.template.Template(filename=input_filename)
+ output_filename = os.path.join( 'spec', 'arb_shader_precision',
+ '{0}-{1}-{2}.shader_test'.format(
+ shader_stage, signature.name,
+ '-'.join(str(argtype)
+ for argtype in signature.argtypes)))
+ print(output_filename)
+ dirname = os.path.dirname(output_filename)
+ if not os.path.exists(dirname):
+ os.makedirs(dirname)
+ with open(output_filename, 'w') as f:
+ f.write(template.render_unicode( signature=signature,
+ test_vectors=test_vectors,
+ tolerances=tolerances,
+ invocation=invocation,
+ num_elements=num_elements,
+ indexers=indexers,
+ shader_runner_type=shader_runner_type,
+ shader_runner_format=shader_runner_format,
+ column_major_values=column_major_values ))
+
+if __name__ == "__main__":
+ main()
diff --git a/generated_tests/shader_precision_templates/fs.mako b/generated_tests/shader_precision_templates/fs.mako
new file mode 100644
index 0000000..aa843b9
--- /dev/null
+++ b/generated_tests/shader_precision_templates/fs.mako
@@ -0,0 +1,127 @@
+[require]
+GLSL >= 4.00
+
+[vertex shader]
+attribute vec4 piglit_vertex;
+void main()
+{
+ gl_Position = piglit_vertex;
+}
+
+[fragment shader]
+% if signature.extension:
+#extension GL_${signature.extension} : require
+% endif
+% for i, arg in enumerate(signature.argtypes):
+uniform ${arg} arg${i};
+% endfor
+uniform float tolerance;
+uniform ${signature.rettype} expected;
+
+void main()
+{
+ ##
+ ## perform the operation being tested
+ ##
+ ${signature.rettype} result = ${invocation};
+ ##
+ ## compare the result(s) to the expected value(s)
+ ##
+ % if signature.rettype.is_matrix or signature.rettype.is_vector:
+ ##
+ ## build an array of bit-level representations of the floating point results calculated above
+ ##
+ int resultbits[${num_elements}] = int[${num_elements}](\
+${', '.join('floatBitsToInt(result{})'.format(i) for i in indexers)}\
+);
+ ##
+ ## build an array of bit-level representations of the passed-in floating point expected results
+ ##
+ int expectedbits[${num_elements}] = int[${num_elements}](\
+${', '.join('floatBitsToInt(expected{})'.format(i) for i in indexers)}\
+);
+ ##
+ ## check for differences in the sign bit for each result
+ ##
+ bool signerr = \
+${' || '.join('(resultbits[{0}]>>31 != expectedbits[{0}]>>31)'.format(i) for i in xrange(0, num_elements))}\
+;
+ ##
+ ## calculate the difference between the generated value and the expected value in ulps
+ ##
+ ${signature.rettype} ulps = ${signature.rettype}(\
+${', '.join('abs(resultbits[{0}] - expectedbits[{0}])'.format(i) for i in xrange(0, num_elements))}\
+);
+ ##
+ ## find the maximum error in ulps of all the calculations using a nested max() sort
+ ##
+ float max_error = \
+ ## start with the outermost max() if there are more than 2 elements
+ ## (two element arrays, eg. vec2, are handled by the final max() below, only)
+ % if num_elements > 2:
+max( \
+ % endif
+ ## cat each value to compare, with an additional nested max() up until the final two values
+ % for i, indexer in enumerate(indexers[:len(indexers)-2]):
+ulps${indexer}, \
+ % if i != len(indexers)-3:
+max(\
+ % endif
+ ## cat the final, deepest, max comparison
+ % endfor
+max(ulps${indexers[len(indexers)-2]}, ulps${indexers[len(indexers)-1]})\
+ ## fill in completing parens
+ % for i in xrange(0, num_elements-2):
+)\
+ % endfor
+;
+ % else:
+ ##
+ ## if there is only a single result value generated, compare it directly
+ ##
+ int resultbits = floatBitsToInt(result);
+ int expectedbits = floatBitsToInt(expected);
+ bool signerr = resultbits>>31 != expectedbits>>31;
+ % if signature.name != 'distance':
+ float ulps = distance(resultbits, expectedbits);
+ % else:
+ float ulps = abs(resultbits - expectedbits);
+ % endif
+ % endif
+ ##
+ ## the test passes if there were no sign errors and the ulps are within tolerance
+ ##
+ gl_FragColor = \
+ % if signature.rettype.is_matrix or signature.rettype.is_vector:
+!signerr && max_error <= tolerance\
+ % else:
+!signerr && ulps <= tolerance\
+ % endif
+ ? vec4(0.0, 1.0, 0.0, 1.0) : vec4(1.0, 0.0, 0.0, 1.0);
+}
+
+% if signature.version_introduced >= 140:
+[vertex data]
+piglit_vertex/float/2
+-1.0 -1.0
+ 1.0 -1.0
+ 1.0 1.0
+-1.0 1.0
+% endif\
+
+[test]
+% for test_num, test_vector in enumerate(test_vectors):
+ % for i in xrange(len(test_vector.arguments)):
+uniform ${shader_runner_type(signature.argtypes[i])} arg${i} ${shader_runner_format( column_major_values(test_vector.arguments[i]))}
+ % endfor
+uniform ${shader_runner_type(signature.rettype)} expected ${shader_runner_format(column_major_values(test_vector.result))}
+uniform float tolerance \
+${tolerances.get(signature.name, 0.0)}
+ % if signature.version_introduced >= 140:
+draw arrays GL_TRIANGLE_FAN 0 4
+ % else:
+draw rect -1 -1 2 2
+ % endif
+## shader_runner uses a 250x250 window so we must ensure that test_num <= 250.
+probe rgba ${test_num % 250} 0 0.0 1.0 0.0 1.0
+% endfor
diff --git a/generated_tests/shader_precision_templates/gs.mako b/generated_tests/shader_precision_templates/gs.mako
new file mode 100644
index 0000000..d512c30
--- /dev/null
+++ b/generated_tests/shader_precision_templates/gs.mako
@@ -0,0 +1,140 @@
+[require]
+GLSL >= 4.00
+
+[vertex shader]
+in vec4 piglit_vertex;
+varying vec4 vertex_to_gs;
+void main()
+{
+ vertex_to_gs = piglit_vertex;
+}
+
+[geometry shader]
+% if signature.extension:
+#extension GL_${signature.extension} : require
+% endif
+layout(triangles) in;
+layout(triangle_strip, max_vertices = 3) out;
+in vec4 vertex_to_gs[3];
+out vec4 color;
+% for i, arg in enumerate(signature.argtypes):
+uniform ${arg} arg${i};
+% endfor
+uniform float tolerance;
+uniform ${signature.rettype} expected;
+
+void main()
+{
+ vec4 tmp_color;
+ ##
+ ## perform the operation being tested
+ ##
+ ${signature.rettype} result = ${invocation};
+ ##
+ ## compare the result(s) to the expected value(s)
+ ##
+ % if signature.rettype.is_matrix or signature.rettype.is_vector:
+ ##
+ ## build an array of bit-level representations of the floating point results calculated above
+ ##
+ int resultbits[${num_elements}] = int[${num_elements}](\
+${', '.join('floatBitsToInt(result{})'.format(i) for i in indexers)}\
+);
+ ##
+ ## build an array of bit-level representations of the passed-in floating point expected results
+ ##
+ int expectedbits[${num_elements}] = int[${num_elements}](\
+${', '.join('floatBitsToInt(expected{})'.format(i) for i in indexers)}\
+);
+ ##
+ ## check for differences in the sign bit for each result
+ ##
+ bool signerr = \
+${' || '.join('(resultbits[{0}]>>31 != expectedbits[{0}]>>31)'.format(i) for i in xrange(0, num_elements))}\
+;
+ ##
+ ## calculate the difference between the generated value and the expected value in ulps
+ ##
+ ${signature.rettype} ulps = ${signature.rettype}(\
+${', '.join('abs(resultbits[{0}] - expectedbits[{0}])'.format(i) for i in xrange(0, num_elements))}\
+);
+ ##
+ ## find the maximum error in ulps of all the calculations using a nested max() sort
+ ##
+ float max_error = \
+ ## start with the outermost max() if there are more than 2 elements
+ ## (two element arrays, eg. vec2, are handled by the final max() below, only)
+ % if num_elements > 2:
+max( \
+ % endif
+ ## cat each value to compare, with an additional nested max() up until the final two values
+ % for i, indexer in enumerate(indexers[:len(indexers)-2]):
+ulps${indexer}, \
+ % if i != len(indexers)-3:
+max(\
+ % endif
+ ## cat the final, deepest, max comparison
+ % endfor
+max(ulps${indexers[len(indexers)-2]}, ulps${indexers[len(indexers)-1]})\
+ ## fill in completing parens
+ % for i in xrange(0, num_elements-2):
+)\
+ % endfor
+;
+ % else:
+ ##
+ ## if there is only a single result value generated, compare it directly
+ ##
+ int resultbits = floatBitsToInt(result);
+ int expectedbits = floatBitsToInt(expected);
+ bool signerr = resultbits>>31 != expectedbits>>31;
+ % if signature.name != 'distance':
+ float ulps = distance(resultbits, expectedbits);
+ % else:
+ float ulps = abs(resultbits - expectedbits);
+ % endif
+ % endif
+ ##
+ ## the test passes if there were no sign errors and the ulps are within tolerance
+ ##
+ tmp_color = \
+ % if signature.rettype.is_matrix or signature.rettype.is_vector:
+!signerr && max_error <= tolerance\
+ % else:
+!signerr && ulps <= tolerance\
+ % endif
+ ? vec4(0.0, 1.0, 0.0, 1.0) : vec4(1.0, 0.0, 0.0, 1.0);
+ for (int i = 0; i < 3; i++) {
+ gl_Position = vertex_to_gs[i];
+ color = tmp_color;
+ EmitVertex();
+ }
+}
+
+[fragment shader]
+varying vec4 color;
+
+void main()
+{
+ gl_FragColor = color;
+}
+
+[vertex data]
+piglit_vertex/float/2
+-1.0 -1.0
+ 1.0 -1.0
+ 1.0 1.0
+-1.0 1.0
+
+[test]
+% for test_num, test_vector in enumerate(test_vectors):
+ % for i in xrange(len(test_vector.arguments)):
+uniform ${shader_runner_type(signature.argtypes[i])} arg${i} ${shader_runner_format( column_major_values(test_vector.arguments[i]))}
+ % endfor
+uniform ${shader_runner_type(signature.rettype)} expected ${shader_runner_format(column_major_values(test_vector.result))}
+uniform float tolerance \
+${tolerances.get(signature.name, 0.0)}
+draw arrays GL_TRIANGLE_FAN 0 4
+## shader_runner uses a 250x250 window so we must ensure that test_num <= 250.
+probe rgba ${test_num % 250} 0 0.0 1.0 0.0 1.0
+% endfor
diff --git a/generated_tests/shader_precision_templates/vs.mako b/generated_tests/shader_precision_templates/vs.mako
new file mode 100644
index 0000000..24cad70
--- /dev/null
+++ b/generated_tests/shader_precision_templates/vs.mako
@@ -0,0 +1,136 @@
+[require]
+GLSL >= 4.00
+
+[vertex shader]
+% if signature.extension:
+#extension GL_${signature.extension} : require
+% endif
+% if signature.version_introduced >= 140:
+in vec4 piglit_vertex;
+out vec4 color;
+% else:
+attribute vec4 piglit_vertex;
+varying vec4 color;
+%endif
+% for i, arg in enumerate(signature.argtypes):
+uniform ${arg} arg${i};
+% endfor
+uniform float tolerance;
+uniform ${signature.rettype} expected;
+
+void main()
+{
+ gl_Position = piglit_vertex;
+ ##
+ ## perform the operation being tested
+ ##
+ ${signature.rettype} result = ${invocation};
+ ##
+ ## compare the result(s) to the expected value(s)
+ ##
+ % if signature.rettype.is_matrix or signature.rettype.is_vector:
+ ##
+ ## build an array of bit-level representations of the floating point results calculated above
+ ##
+ int resultbits[${num_elements}] = int[${num_elements}](\
+${', '.join('floatBitsToInt(result{})'.format(i) for i in indexers)}\
+);
+ ##
+ ## build an array of bit-level representations of the passed-in floating point expected results
+ ##
+ int expectedbits[${num_elements}] = int[${num_elements}](\
+${', '.join('floatBitsToInt(expected{})'.format(i) for i in indexers)}\
+);
+ ##
+ ## check for differences in the sign bit for each result
+ ##
+ bool signerr = \
+${' || '.join('(resultbits[{0}]>>31 != expectedbits[{0}]>>31)'.format(i) for i in xrange(0, num_elements))}\
+;
+ ##
+ ## calculate the difference between the generated value and the expected value in ulps
+ ##
+ ${signature.rettype} ulps = ${signature.rettype}(\
+${', '.join('abs(resultbits[{0}] - expectedbits[{0}])'.format(i) for i in xrange(0, num_elements))}\
+);
+ ##
+ ## find the maximum error in ulps of all the calculations using a nested max() sort
+ ##
+ float max_error = \
+ ## start with the outermost max() if there are more than 2 elements
+ ## (two element arrays, eg. vec2, are handled by the final max() below, only)
+ % if num_elements > 2:
+max( \
+ % endif
+ ## cat each value to compare, with an additional nested max() up until the final two values
+ % for i, indexer in enumerate(indexers[:len(indexers)-2]):
+ulps${indexer}, \
+ % if i != len(indexers)-3:
+max(\
+ % endif
+ ## cat the final, deepest, max comparison
+ % endfor
+max(ulps${indexers[len(indexers)-2]}, ulps${indexers[len(indexers)-1]})\
+ ## fill in completing parens
+ % for i in xrange(0, num_elements-2):
+)\
+ % endfor
+;
+ % else:
+ ##
+ ## if there is only a single result value generated, compare it directly
+ ##
+ int resultbits = floatBitsToInt(result);
+ int expectedbits = floatBitsToInt(expected);
+ bool signerr = resultbits>>31 != expectedbits>>31;
+ % if signature.name != 'distance':
+ float ulps = distance(resultbits, expectedbits);
+ % else:
+ float ulps = abs(resultbits - expectedbits);
+ % endif
+ % endif
+ ##
+ ## the test passes if there were no sign errors and the ulps are within tolerance
+ ##
+ color = \
+ % if signature.rettype.is_matrix or signature.rettype.is_vector:
+!signerr && max_error <= tolerance\
+ % else:
+!signerr && ulps <= tolerance\
+ % endif
+ ? vec4(0.0, 1.0, 0.0, 1.0) : vec4(1.0, 0.0, 0.0, 1.0);
+}
+
+[fragment shader]
+varying vec4 color;
+
+void main()
+{
+ gl_FragColor = color;
+}
+
+% if signature.version_introduced >= 140:
+[vertex data]
+piglit_vertex/float/2
+-1.0 -1.0
+ 1.0 -1.0
+ 1.0 1.0
+-1.0 1.0
+% endif\
+
+[test]
+% for test_num, test_vector in enumerate(test_vectors):
+ % for i in xrange(len(test_vector.arguments)):
+uniform ${shader_runner_type(signature.argtypes[i])} arg${i} ${shader_runner_format( column_major_values(test_vector.arguments[i]))}
+ % endfor
+uniform ${shader_runner_type(signature.rettype)} expected ${shader_runner_format(column_major_values(test_vector.result))}
+uniform float tolerance \
+${tolerances.get(signature.name, 0.0)}
+ % if signature.version_introduced >= 140:
+draw arrays GL_TRIANGLE_FAN 0 4
+ % else:
+draw rect -1 -1 2 2
+ % endif
+## shader_runner uses a 250x250 window so we must ensure that test_num <= 250.
+probe rgba ${test_num % 250} 0 0.0 1.0 0.0 1.0
+% endfor
--
2.1.3
More information about the Piglit
mailing list