[Mesa-dev] Improving precision of mod(x,y)

[Iago Toral]

Hi,

We have 16 deqp tests that fail, at least on i965, because of
insufficient precision of the mod GLSL function.

Mesa lowers mod(x,y) to y * fract(x,y) so there can be some precision
lost due to fract operation. Since the result is multiplied by y the
total precision lost usually grows together with the value of y.

Below are some examples to give an idea of the magnitude of this error.
The values on the right represent the precision error for each case:

mod(-1.951171875, 1.9980468750) =>  0.0000000447
mod(121.57, 13.29)              =>  0.0000023842
mod(3769.12, 321.99)            =>  0.0000762939
mod(3769.12, 1321.99)           =>  0.0001220703
mod(-987654.125, 123456.984375) =>  0.0160663128
mod( 987654.125, 123456.984375) =>  0.0312500000

As you see, for large enough values, the precision error becomes
significant.

This can be fixed by lowering mod(x,y) to x - y * floor(x/y) instead,
which is the suggested implementation in the GLSL docs. I have a local
patch in my tree that does this and it does indeed fix the problem. the
down side is that this implementation adds and extra ADD instruction to
the generated code (besides replacing fract with floor, which I guess
have similar cost).

Since this is a case where there is some trade-off to the fix, I wonder
if we are interested in doing this or not. Is the precision fix worth
the additional ADD?

Iago