[Mesa-dev] About tolerance calculation on specific (builtin) functions
agomez at igalia.com
Wed May 4 17:05:12 UTC 2016
as part of the work done to "Add FP64 support to the i965 shader
I've been working to add piglit tests that would check the new features
added by this addition.
Due to this, I've been checking and making modifications into the
builtin_functions*py module used by some generators. These modules use
automatic calculation of the tolerance when distance checking the
result of a function.
As already stated in the good documentation of the module, the
tolerance is computed following what it is stated in OpenGL's specs:
From the OpenGL 1.4 spec (2.1.1 "Floating-Point Computation"):
"We require simply that numbers' floating-point parts contain
enough bits ... so that individual results of floating-point
operations are accurate to about 1 part in 10^5."
Although the text is open to interpretation, and for specific
operations we take a little bit more flexible approach, basically, the
tolerance is calculated as:
tolerance = <expected_value> / 10⁵
This makes sense since the precision of a floating point value gets
reduced while the number gets bigger.
Following this approach, for a number in the order of 40*10⁵, the
tolerance used is ~40. While this should be OK for most of the
functions, it seems to me that such a high tolerance should not be used
with specific functions, if any tolerance should be used at all.
For example, when testing the "sign()" function, seems pretty obvious
that using a value of 40 in the tolerance of a function that should
return either 1.0, 0.0 or -1.0 doesn't make much sense.
A similar case is the "trunc" function and probably others, like
"floor", "ceil", "abs", etc.
My conclusion is that it should be safe to assume no tolerance in this
functions and I could modify the algorithm used for them in the python
module but I wanted to have some feedback in case I'm not taking into
account something that advices against doing these modifications.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 181 bytes
Desc: This is a digitally signed message part
More information about the mesa-dev