[Piglit] [PATCH 0/1] Add comprehensive tests of builtin functions with uniform input.

[Paul Berry]

On 23 July 2011 01:45, Ian Romanick <idr at freedesktop.org> wrote:
>> [test]
>> uniform float arg0 -1.9
>> uniform float arg1 -0.633333333333
>> uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
>> draw rect -1 -1 2 2
>> probe rgba 0 0 0.25 0.748979443068 0.75 0.75
>
> Checks like this are a big problem for shaders.  With 8-bit color
> outputs, checks that require any precision are doomed.  As you noticed,
> you can only validate answers on the range [0,1].  Moreover, you can
> only verify that the calculated answer is within 1/255 of correct.
> That's just not good enough for most things.

That's a really good point.  As long as we are going to all this
effort to comprehensively test all the built-in functions, we really
should test them to a reasonable precision.

>
> Most of the time you want to check that a calculation is correct within
> a certain tolerance.  In those cases, you should do exactly that in the
> shader.  Some of the non-constant indexing tests that I wrote do this.
> In one of the cases, vs-varying-mat4-wr.shader_test, I want to check
> that a matrix passed to fragment shader has a varying had correct
> values.  I checked that by doing a calculation with that matrix and
> verifying the correctness of that result:
>
> void main()
> {
>    gl_FragColor = (abs(distance(dst_matrix * v, expect)) < 1e-6)
>        ? vec4(0.0, 1.0, 0.0, 1.0) : vec4(1.0, 0.0, 0.0, 1.0);
> }

I think it would be fairly trivial to change
gen_builtin_uniform_tests.py to generate code like this.  I'll submit
a follow-up patch.

>
> Of course, this assumes that abs() and distance() work correctly. :)

Actually the call to abs() could be dropped, since distance() always
returns a nonnegative result.  But your point remains.  If you want to
be pedantic about it (and who doesn't?) we're also assuming that <,
?:, vec4(), gl_FragColor, and floating point constants work correctly
:)

I think the right balance between caution and solipsism here is to
drop the call to abs(), and do a little extra work in the distance()
test so that it's not begging the question.  I'll try to do this in my
follow-up patch.

In your test, how did you decide on 1e-6 as the tolerance?  According
to GLSL 1.20 (in section 4.1.4 "Floats"), "It is not required that the
precision of internal processing match the IEEE floating-point
specification for floating-point operations, but the guidelines for
precision established by the OpenGL 1.4 specification must be met."  I
don't have the OpenGL 1.4 spec handy, but OpenGL 2.1 says (in section
2.1.1 "Floating-Point Computation") "We require simply that numbers'
floating-point parts contain enough bits ... so that individual
results of floating-point operations are accurate to about 1 part in
10^5."  I haven't checked whether later versions of OpenGL and GLSL
impose stricter precision requirements.

It seems to me that this means we should use a tolerance of at least
1e-5, and sometimes more than that (in cases like degrees(), where the
derivative of the output with respect to the input is substantially
greater than 1).  I'll mess around with builtin_function.py and see
how difficult it would be to make it compute an appropriate tolerance
for every test vector.