[Piglit] [PATCH] arb_shader_precision: add tests for floating point precision

Micah Fedke micah.fedke at collabora.co.uk
Fri Dec 19 09:01:05 PST 2014


This generated_tests script creates a suite of tests that measure the
floating point precision of most GLSL built-ins, according to
ARB_shader_precision.  Test vectors come from builtin_function.py, but
are filtered down to avoid non-float types.

These tests are reporting precision errors in ceil, cross, degrees,
distance, inverse, mix, mod, normalize, op-assign-div, op-assign-mult,
op-div, op-mult, reflect, refract and smoothstep on Ivybridge.

---

Updates since previous (RFC v5):

RFC ended, now submitting as a patch

Suppressed the use of repr() in the generator
  - python's repr() does not output enough digits to uniquely
    represent a 32 bit float value - the {0:1.8e} format is 
    used instead

Eliminated the use of distance() for ulps calculations
  - this was producing an imprecise result in some situations

Updated the list of failing functions in the description


Note: I am new to the project and don't have commit access.

 generated_tests/CMakeLists.txt                     |  10 +-
 generated_tests/gen_shader_precision_tests.py      | 159 +++++++++++++++++++++
 generated_tests/shader_precision_templates/fs.mako | 127 ++++++++++++++++
 generated_tests/shader_precision_templates/gs.mako | 140 ++++++++++++++++++
 generated_tests/shader_precision_templates/vs.mako | 136 ++++++++++++++++++
 5 files changed, 571 insertions(+), 1 deletion(-)
 create mode 100644 generated_tests/gen_shader_precision_tests.py
 create mode 100644 generated_tests/shader_precision_templates/fs.mako
 create mode 100644 generated_tests/shader_precision_templates/gs.mako
 create mode 100644 generated_tests/shader_precision_templates/vs.mako

diff --git a/generated_tests/CMakeLists.txt b/generated_tests/CMakeLists.txt
index 6d27b3e..77ee06c 100644
--- a/generated_tests/CMakeLists.txt
+++ b/generated_tests/CMakeLists.txt
@@ -91,6 +91,13 @@ piglit_make_generated_tests(
 	constant_array_size_tests_fp64.list
 	gen_constant_array_size_tests_fp64.py
 	builtin_function_fp64.py)
+piglit_make_generated_tests(
+	shader_precision_tests.list
+	gen_shader_precision_tests.py
+	builtin_function.py
+	shader_precision_templates/vs.mako
+	shader_precision_templates/fs.mako
+	shader_precision_templates/gs.mako)
 
 # Add a "gen-tests" target that can be used to generate all the
 # tests without doing any other compilation.
@@ -113,4 +120,5 @@ add_custom_target(gen-tests ALL
 		uniform-initializer_tests.list
 		interpolation-qualifier-built-in-variable.list
 		builtin_uniform_tests_fp64.list
-		constant_array_size_tests_fp64.list)
+		constant_array_size_tests_fp64.list
+		shader_precision_tests.list)
diff --git a/generated_tests/gen_shader_precision_tests.py b/generated_tests/gen_shader_precision_tests.py
new file mode 100644
index 0000000..2eab2b0
--- /dev/null
+++ b/generated_tests/gen_shader_precision_tests.py
@@ -0,0 +1,159 @@
+# coding=utf-8
+#
+# Copyright © 2014 Intel Corporation
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice (including the next
+# paragraph) shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+# DEALINGS IN THE SOFTWARE.
+
+
+"""Generate a set of shader_runner tests for overloaded versions of every
+ built-in function specified in arb_shader_precision, based on the test
+ vectors computed by builtin_function.py, and structured according to the mako
+ templates in shader_precision_templates/.
+
+ The vertex, geometry, and fragment shader types are exercised by each test.
+ In all cases, the inputs to the built-in functions come from uniforms, so
+ that the effectiveness of the test won't be circumvented by constant folding
+ in the GLSL compiler.
+
+ The tests operate by invoking the built-in function in the appropriate
+ shader, calculating any deviance from the expected value (in ulps), comparing
+ the deviance to a supplied tolerance (according to those specified in
+ arb_shader_precision), and then outputting the pass/fail result as a solid
+ rgba color which is then checked using shader_runner's "probe rgba" command.
+
+ For built-in functions whose result type is multi-valued (vec or mat), the
+ tests calculate the error in ulps for each element separately, but compare
+ only the largest error value to the tolerance.  This accounts for cases where
+ error varies among the elements of a vec or mat result.
+
+ This program outputs, to stdout, the name of each file it generates.
+"""
+
+from builtin_function import * 
+import mako.template 
+import os 
+import os.path
+
+tolerances = {'pow': 16.0, 
+              'exp': 3.0,
+              'exp2': 3.0,
+              'log': 3.0,
+              'log2': 3.0,
+              'sqrt': 3.0,
+              'inversesqrt': 2.0}
+
+trig_builtins = ('sin', 'cos', 'tan', 
+                 'asin', 'acos', 'atan', 
+                 'sinh', 'cosh', 'tanh', 
+                 'asinh', 'acosh', 'atanh')
+
+def make_indexers(signature):
+   """Build a list of strings which index into every possible
+   value of the result.  For example, if the result is a vec2,
+   then build the indexers ['[0]', '[1]'].
+   """
+   if signature.rettype.num_cols == 1:
+      col_indexers = ['']
+   else:
+      col_indexers = ['[{0}]'.format(i) for i in xrange(signature.rettype.num_cols)]
+   if signature.rettype.num_rows == 1:
+      row_indexers = ['']
+   else:
+      row_indexers = ['[{0}]'.format(i) for i in xrange(signature.rettype.num_rows)]
+   return [col_indexer + row_indexer 
+           for col_indexer in col_indexers for row_indexer in row_indexers]
+
+def shader_runner_type(glsl_type):
+    """Return the appropriate type name necessary for binding a
+    uniform of the given type using shader_runner's "uniform" command.
+    Boolean values and vectors are converted to ints, and square
+    matrices are written in "matNxN" form.
+    """
+    if glsl_type.base_type == glsl_bool:
+        if glsl_type.is_scalar:
+            return 'int'
+        else:
+            return 'ivec{0}'.format(glsl_type.num_rows)
+    elif glsl_type.is_matrix:
+        return 'mat{0}x{1}'.format(glsl_type.num_cols, glsl_type.num_rows)
+    else:
+        return str(glsl_type)
+
+def shader_runner_format(values):
+    """Format the given values for use in a shader_runner "uniform" or
+    "probe rgba" command.  Bools are converted to 0's and 1's, and
+    values are separated by spaces.
+    """
+    transformed_values = []
+    retval = ''
+    for value in values:
+        if isinstance(value, (bool, np.bool_)):
+            transformed_values.append(int(value))
+        else:
+            transformed_values.append(value)
+    for x in transformed_values:
+        if isinstance(x,np.float32):
+            retval+=' {0}'.format('{0:1.8e}'.format(x))
+        else:
+            retval+=' {0}'.format(repr(x))
+    return retval
+
+def main():
+    """ Main function """
+
+    for signature, test_vectors in sorted(test_suite.iteritems()):
+        arg_float_check = tuple(
+                        arg.base_type == glsl_float for arg in signature.argtypes)
+        # Filter the test vectors down to only those which deal exclusively in float types
+        #and are not trig functions or determinant()
+        indexers = make_indexers(signature)
+        num_elements = signature.rettype.num_cols*signature.rettype.num_rows
+        invocation = signature.template.format( *['arg{0}'.format(i) 
+                                                for i in xrange(len(signature.argtypes))])
+        if (signature.rettype.base_type == glsl_float and
+            arg_float_check and
+            all(arg_float_check) and
+            signature.name not in trig_builtins and
+            signature.name != 'determinant'): 
+            for shader_stage in ('vs', 'fs', 'gs'):
+                input_filename = 'shader_precision_templates/{0}.mako'.format(shader_stage)
+                template = mako.template.Template(filename=input_filename)
+                output_filename = os.path.join( 'spec', 'arb_shader_precision',
+                                                '{0}-{1}-{2}.shader_test'.format(
+                                                shader_stage, signature.name, 
+                                                '-'.join(str(argtype) 
+                                                for argtype in signature.argtypes)))
+                print(output_filename)
+                dirname = os.path.dirname(output_filename)
+                if not os.path.exists(dirname):
+                    os.makedirs(dirname)
+                with open(output_filename, 'w') as f:
+                    f.write(template.render_unicode( signature=signature, 
+                                                     test_vectors=test_vectors,
+                                                     tolerances=tolerances,
+                                                     invocation=invocation,
+                                                     num_elements=num_elements,
+                                                     indexers=indexers,
+                                                     shader_runner_type=shader_runner_type,
+                                                     shader_runner_format=shader_runner_format,
+                                                     column_major_values=column_major_values ))
+
+if __name__ == "__main__":
+    main()
diff --git a/generated_tests/shader_precision_templates/fs.mako b/generated_tests/shader_precision_templates/fs.mako
new file mode 100644
index 0000000..aa843b9
--- /dev/null
+++ b/generated_tests/shader_precision_templates/fs.mako
@@ -0,0 +1,127 @@
+[require]
+GLSL >= 4.00
+
+[vertex shader]
+attribute vec4 piglit_vertex;
+void main()
+{
+        gl_Position = piglit_vertex;
+}
+
+[fragment shader]
+% if signature.extension:
+#extension GL_${signature.extension} : require
+% endif
+% for i, arg in enumerate(signature.argtypes):
+uniform ${arg} arg${i};
+% endfor
+uniform float tolerance;
+uniform ${signature.rettype} expected;
+
+void main()
+{
+  ##
+  ## perform the operation being tested
+  ##
+  ${signature.rettype} result = ${invocation};
+  ##
+  ## compare the result(s) to the expected value(s)
+  ##
+  % if signature.rettype.is_matrix or signature.rettype.is_vector:
+    ##
+    ## build an array of bit-level representations of the floating point results calculated above
+    ##
+  int resultbits[${num_elements}] = int[${num_elements}](\
+${', '.join('floatBitsToInt(result{})'.format(i) for i in indexers)}\
+);
+    ##
+    ## build an array of bit-level representations of the passed-in floating point expected results
+    ##
+  int expectedbits[${num_elements}] = int[${num_elements}](\
+${', '.join('floatBitsToInt(expected{})'.format(i) for i in indexers)}\
+);
+  ##
+  ## check for differences in the sign bit for each result
+  ##
+  bool signerr = \
+${' || '.join('(resultbits[{0}]>>31 != expectedbits[{0}]>>31)'.format(i) for i in xrange(0, num_elements))}\
+;
+  ##
+  ## calculate the difference between the generated value and the expected value in ulps
+  ##
+  ${signature.rettype} ulps = ${signature.rettype}(\
+${', '.join('abs(resultbits[{0}] - expectedbits[{0}])'.format(i) for i in xrange(0, num_elements))}\
+);
+  ##
+  ## find the maximum error in ulps of all the calculations using a nested max() sort
+  ##
+  float max_error = \
+    ## start with the outermost max() if there are more than 2 elements
+    ## (two element arrays, eg. vec2, are handled by the final max() below, only)
+    % if num_elements > 2:
+max( \
+    % endif
+    ## cat each value to compare, with an additional nested max() up until the final two values
+    % for i, indexer in enumerate(indexers[:len(indexers)-2]):
+ulps${indexer}, \
+    % if i != len(indexers)-3:
+max(\
+    % endif
+    ## cat the final, deepest, max comparison
+    % endfor
+max(ulps${indexers[len(indexers)-2]}, ulps${indexers[len(indexers)-1]})\
+    ## fill in completing parens
+    % for i in xrange(0, num_elements-2):
+)\
+    % endfor
+;
+  % else:
+    ##
+    ## if there is only a single result value generated, compare it directly
+    ##
+  int resultbits = floatBitsToInt(result);
+  int expectedbits = floatBitsToInt(expected);
+  bool signerr = resultbits>>31 != expectedbits>>31;
+    % if signature.name != 'distance':
+  float ulps = distance(resultbits, expectedbits);
+    % else:
+  float ulps = abs(resultbits - expectedbits);
+    % endif
+  % endif
+  ##
+  ## the test passes if there were no sign errors and the ulps are within tolerance
+  ##
+  gl_FragColor = \
+  % if signature.rettype.is_matrix or signature.rettype.is_vector:
+!signerr && max_error <= tolerance\
+  % else:
+!signerr && ulps <= tolerance\
+  % endif
+ ? vec4(0.0, 1.0, 0.0, 1.0) : vec4(1.0, 0.0, 0.0, 1.0);
+}
+
+% if signature.version_introduced >= 140:
+[vertex data]
+piglit_vertex/float/2
+-1.0 -1.0
+ 1.0 -1.0
+ 1.0  1.0
+-1.0  1.0
+% endif\
+
+[test]
+% for test_num, test_vector in enumerate(test_vectors):
+  % for i in xrange(len(test_vector.arguments)):
+uniform ${shader_runner_type(signature.argtypes[i])} arg${i} ${shader_runner_format( column_major_values(test_vector.arguments[i]))}
+  % endfor
+uniform ${shader_runner_type(signature.rettype)} expected ${shader_runner_format(column_major_values(test_vector.result))}
+uniform float tolerance \
+${tolerances.get(signature.name, 0.0)}
+  % if signature.version_introduced >= 140:
+draw arrays GL_TRIANGLE_FAN 0 4
+  % else:
+draw rect -1 -1 2 2
+  % endif
+## shader_runner uses a 250x250 window so we must ensure that test_num <= 250.
+probe rgba ${test_num % 250} 0 0.0 1.0 0.0 1.0
+% endfor
diff --git a/generated_tests/shader_precision_templates/gs.mako b/generated_tests/shader_precision_templates/gs.mako
new file mode 100644
index 0000000..d512c30
--- /dev/null
+++ b/generated_tests/shader_precision_templates/gs.mako
@@ -0,0 +1,140 @@
+[require]
+GLSL >= 4.00
+
+[vertex shader]
+in vec4 piglit_vertex;
+varying vec4 vertex_to_gs;
+void main()
+{
+     vertex_to_gs = piglit_vertex;
+}
+
+[geometry shader]
+% if signature.extension:
+#extension GL_${signature.extension} : require
+% endif
+layout(triangles) in;
+layout(triangle_strip, max_vertices = 3) out;
+in vec4 vertex_to_gs[3];
+out vec4 color;
+% for i, arg in enumerate(signature.argtypes):
+uniform ${arg} arg${i};
+% endfor
+uniform float tolerance;
+uniform ${signature.rettype} expected;
+
+void main()
+{
+  vec4 tmp_color;
+  ##
+  ## perform the operation being tested
+  ##
+  ${signature.rettype} result = ${invocation};
+  ##
+  ## compare the result(s) to the expected value(s)
+  ##
+  % if signature.rettype.is_matrix or signature.rettype.is_vector:
+    ##
+    ## build an array of bit-level representations of the floating point results calculated above
+    ##
+  int resultbits[${num_elements}] = int[${num_elements}](\
+${', '.join('floatBitsToInt(result{})'.format(i) for i in indexers)}\
+);
+    ##
+    ## build an array of bit-level representations of the passed-in floating point expected results
+    ##
+  int expectedbits[${num_elements}] = int[${num_elements}](\
+${', '.join('floatBitsToInt(expected{})'.format(i) for i in indexers)}\
+);
+  ##
+  ## check for differences in the sign bit for each result
+  ##
+  bool signerr = \
+${' || '.join('(resultbits[{0}]>>31 != expectedbits[{0}]>>31)'.format(i) for i in xrange(0, num_elements))}\
+;
+  ##
+  ## calculate the difference between the generated value and the expected value in ulps
+  ##
+  ${signature.rettype} ulps = ${signature.rettype}(\
+${', '.join('abs(resultbits[{0}] - expectedbits[{0}])'.format(i) for i in xrange(0, num_elements))}\
+);
+  ##
+  ## find the maximum error in ulps of all the calculations using a nested max() sort
+  ##
+  float max_error = \
+    ## start with the outermost max() if there are more than 2 elements
+    ## (two element arrays, eg. vec2, are handled by the final max() below, only)
+    % if num_elements > 2:
+max( \
+    % endif
+    ## cat each value to compare, with an additional nested max() up until the final two values
+    % for i, indexer in enumerate(indexers[:len(indexers)-2]):
+ulps${indexer}, \
+    % if i != len(indexers)-3:
+max(\
+    % endif
+    ## cat the final, deepest, max comparison
+    % endfor
+max(ulps${indexers[len(indexers)-2]}, ulps${indexers[len(indexers)-1]})\
+    ## fill in completing parens
+    % for i in xrange(0, num_elements-2):
+)\
+    % endfor
+;
+  % else:
+    ##
+    ## if there is only a single result value generated, compare it directly
+    ##
+  int resultbits = floatBitsToInt(result);
+  int expectedbits = floatBitsToInt(expected);
+  bool signerr = resultbits>>31 != expectedbits>>31;
+    % if signature.name != 'distance':
+  float ulps = distance(resultbits, expectedbits);
+    % else:
+  float ulps = abs(resultbits - expectedbits);
+    % endif
+  % endif
+  ##
+  ## the test passes if there were no sign errors and the ulps are within tolerance
+  ##
+  tmp_color = \
+  % if signature.rettype.is_matrix or signature.rettype.is_vector:
+!signerr && max_error <= tolerance\
+  % else:
+!signerr && ulps <= tolerance\
+  % endif
+ ? vec4(0.0, 1.0, 0.0, 1.0) : vec4(1.0, 0.0, 0.0, 1.0);
+  for (int i = 0; i < 3; i++) {
+    gl_Position = vertex_to_gs[i];
+    color = tmp_color;
+    EmitVertex();
+  }
+}
+
+[fragment shader]
+varying vec4 color;
+
+void main()
+{
+  gl_FragColor = color;
+}
+
+[vertex data]
+piglit_vertex/float/2
+-1.0 -1.0
+ 1.0 -1.0
+ 1.0  1.0
+-1.0  1.0
+
+[test]
+% for test_num, test_vector in enumerate(test_vectors):
+  % for i in xrange(len(test_vector.arguments)):
+uniform ${shader_runner_type(signature.argtypes[i])} arg${i} ${shader_runner_format( column_major_values(test_vector.arguments[i]))}
+  % endfor
+uniform ${shader_runner_type(signature.rettype)} expected ${shader_runner_format(column_major_values(test_vector.result))}
+uniform float tolerance \
+${tolerances.get(signature.name, 0.0)}
+draw arrays GL_TRIANGLE_FAN 0 4
+## shader_runner uses a 250x250 window so we must ensure that test_num <= 250.
+probe rgba ${test_num % 250} 0 0.0 1.0 0.0 1.0
+% endfor
diff --git a/generated_tests/shader_precision_templates/vs.mako b/generated_tests/shader_precision_templates/vs.mako
new file mode 100644
index 0000000..24cad70
--- /dev/null
+++ b/generated_tests/shader_precision_templates/vs.mako
@@ -0,0 +1,136 @@
+[require]
+GLSL >= 4.00
+
+[vertex shader]
+% if signature.extension:
+#extension GL_${signature.extension} : require
+% endif
+% if signature.version_introduced >= 140:
+in vec4 piglit_vertex;
+out vec4 color;
+% else:
+attribute vec4 piglit_vertex;
+varying vec4 color;
+%endif
+% for i, arg in enumerate(signature.argtypes):
+uniform ${arg} arg${i};
+% endfor
+uniform float tolerance;
+uniform ${signature.rettype} expected;
+
+void main()
+{
+  gl_Position = piglit_vertex;
+  ##
+  ## perform the operation being tested
+  ##
+  ${signature.rettype} result = ${invocation};
+  ##
+  ## compare the result(s) to the expected value(s)
+  ##
+  % if signature.rettype.is_matrix or signature.rettype.is_vector:
+    ##
+    ## build an array of bit-level representations of the floating point results calculated above
+    ##
+  int resultbits[${num_elements}] = int[${num_elements}](\
+${', '.join('floatBitsToInt(result{})'.format(i) for i in indexers)}\
+);
+    ##
+    ## build an array of bit-level representations of the passed-in floating point expected results
+    ##
+  int expectedbits[${num_elements}] = int[${num_elements}](\
+${', '.join('floatBitsToInt(expected{})'.format(i) for i in indexers)}\
+);
+  ##
+  ## check for differences in the sign bit for each result
+  ##
+  bool signerr = \
+${' || '.join('(resultbits[{0}]>>31 != expectedbits[{0}]>>31)'.format(i) for i in xrange(0, num_elements))}\
+;
+  ##
+  ## calculate the difference between the generated value and the expected value in ulps
+  ##
+  ${signature.rettype} ulps = ${signature.rettype}(\
+${', '.join('abs(resultbits[{0}] - expectedbits[{0}])'.format(i) for i in xrange(0, num_elements))}\
+);
+  ##
+  ## find the maximum error in ulps of all the calculations using a nested max() sort
+  ##
+  float max_error = \
+    ## start with the outermost max() if there are more than 2 elements
+    ## (two element arrays, eg. vec2, are handled by the final max() below, only)
+    % if num_elements > 2:
+max( \
+    % endif
+    ## cat each value to compare, with an additional nested max() up until the final two values
+    % for i, indexer in enumerate(indexers[:len(indexers)-2]):
+ulps${indexer}, \
+    % if i != len(indexers)-3:
+max(\
+    % endif
+    ## cat the final, deepest, max comparison
+    % endfor
+max(ulps${indexers[len(indexers)-2]}, ulps${indexers[len(indexers)-1]})\
+    ## fill in completing parens
+    % for i in xrange(0, num_elements-2):
+)\
+    % endfor
+;
+  % else:
+    ##
+    ## if there is only a single result value generated, compare it directly
+    ##
+  int resultbits = floatBitsToInt(result);
+  int expectedbits = floatBitsToInt(expected);
+  bool signerr = resultbits>>31 != expectedbits>>31;
+    % if signature.name != 'distance':
+  float ulps = distance(resultbits, expectedbits);
+    % else:
+  float ulps = abs(resultbits - expectedbits);
+    % endif
+  % endif
+  ##
+  ## the test passes if there were no sign errors and the ulps are within tolerance
+  ##
+  color = \
+  % if signature.rettype.is_matrix or signature.rettype.is_vector:
+!signerr && max_error <= tolerance\
+  % else:
+!signerr && ulps <= tolerance\
+  % endif
+ ? vec4(0.0, 1.0, 0.0, 1.0) : vec4(1.0, 0.0, 0.0, 1.0);
+}
+
+[fragment shader]
+varying vec4 color;
+
+void main()
+{
+  gl_FragColor = color;
+}
+
+% if signature.version_introduced >= 140:
+[vertex data]
+piglit_vertex/float/2
+-1.0 -1.0
+ 1.0 -1.0
+ 1.0  1.0
+-1.0  1.0
+% endif\
+
+[test]
+% for test_num, test_vector in enumerate(test_vectors):
+  % for i in xrange(len(test_vector.arguments)):
+uniform ${shader_runner_type(signature.argtypes[i])} arg${i} ${shader_runner_format( column_major_values(test_vector.arguments[i]))}
+  % endfor
+uniform ${shader_runner_type(signature.rettype)} expected ${shader_runner_format(column_major_values(test_vector.result))}
+uniform float tolerance \
+${tolerances.get(signature.name, 0.0)}
+  % if signature.version_introduced >= 140:
+draw arrays GL_TRIANGLE_FAN 0 4
+  % else:
+draw rect -1 -1 2 2
+  % endif
+## shader_runner uses a 250x250 window so we must ensure that test_num <= 250.
+probe rgba ${test_num % 250} 0 0.0 1.0 0.0 1.0
+% endfor
-- 
2.1.3



More information about the Piglit mailing list