[Mesa-dev] About tolerance calculation on specific (builtin) functions

[Andres Gomez]

Hi,

as part of the work done to "Add FP64 support to the i965 shader
backends" at:
https://bugs.freedesktop.org/show_bug.cgi?id=92760

I've been working to add piglit tests that would check the new features
added by this addition.

Due to this, I've been checking and making modifications into the
builtin_functions*py module used by some generators. These modules use
automatic calculation of the tolerance when distance checking the
result of a function.

As already stated in the good documentation of the module, the
tolerance is computed following what it is stated in OpenGL's specs:

    From the OpenGL 1.4 spec (2.1.1 "Floating-Point Computation"):

      "We require simply that numbers' floating-point parts contain
      enough bits ... so that individual results of floating-point
      operations are accurate to about 1 part in 10^5."

Although the text is open to interpretation, and for specific
operations we take a little bit more flexible approach, basically, the
tolerance is calculated as:

tolerance = <expected_value> / 10⁵

This makes sense since the precision of a floating point value gets
reduced while the number gets bigger[1].

Following this approach, for a number in the order of 40*10⁵, the
tolerance used is ~40. While this should be OK for most of the
functions, it seems to me that such a high tolerance should not be used
with specific functions, if any tolerance should be used at all.

For example, when testing the "sign()" function, seems pretty obvious
that using a value of 40 in the tolerance of a function that should
return either 1.0, 0.0 or -1.0 doesn't make much sense.

A similar case is the "trunc" function and probably others, like
"floor", "ceil", "abs", etc.

My conclusion is that it should be safe to assume no tolerance in this
functions and I could modify the algorithm used for them in the python
module but I wanted to have some feedback in case I'm not taking into
account something that advices against doing these modifications.

Opinions?

[1] https://en.wikipedia.org/wiki/IEEE_floating_point#Basic_and_interch
ange_formats
-- 
Br,

Andres

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part
URL: <https://lists.freedesktop.org/archives/mesa-dev/attachments/20160504/7446d810/attachment.sig>