[Piglit] [PATCH 0/1] Add comprehensive tests of builtin functions with uniform input.

Paul Berry stereotype441 at gmail.com
Sat Jul 23 16:48:20 PDT 2011


On 23 July 2011 08:32, Eric Anholt <eric at anholt.net> wrote:
> On Sat, 23 Jul 2011 01:45:38 -0700, Ian Romanick <idr at freedesktop.org> wrote:
>> On 07/22/2011 02:00 PM, Paul Berry wrote:
>> > [require]
>> > GLSL >= 1.10
>> >
>> > [vertex shader]
>> > varying vec4 color;
>> > uniform float arg0;
>> > uniform float arg1;
>> > uniform vec4 arg2;
>> >
>> > void main()
>> > {
>> >   gl_Position = gl_Vertex;
>> >   vec4 result = smoothstep(arg0, arg1, arg2);
>> >   result -= -0.5;
>> >   result *= 0.5;
>> >   color = vec4(result);
>> > }
>> >
>> > [fragment shader]
>> > varying vec4 color;
>> >
>> > void main()
>> > {
>> >   gl_FragColor = color;
>> > }
>> >
>> > [test]
>> > uniform float arg0 -1.9
>> > uniform float arg1 -0.633333333333
>> > uniform vec4 arg2 -2.0 -0.666666666667 0.666666666667 2.0
>> > draw rect -1 -1 2 2
>> > probe rgba 0 0 0.25 0.748979443068 0.75 0.75
>>
>> Checks like this are a big problem for shaders.  With 8-bit color
>> outputs, checks that require any precision are doomed.  As you noticed,
>> you can only validate answers on the range [0,1].  Moreover, you can
>> only verify that the calculated answer is within 1/255 of correct.
>> That's just not good enough for most things.
>>
>> Most of the time you want to check that a calculation is correct within
>> a certain tolerance.  In those cases, you should do exactly that in the
>> shader.  Some of the non-constant indexing tests that I wrote do this.
>> In one of the cases, vs-varying-mat4-wr.shader_test, I want to check
>> that a matrix passed to fragment shader has a varying had correct
>> values.  I checked that by doing a calculation with that matrix and
>> verifying the correctness of that result:
>>
>> void main()
>> {
>>     gl_FragColor = (abs(distance(dst_matrix * v, expect)) < 1e-6)
>>         ? vec4(0.0, 1.0, 0.0, 1.0) : vec4(1.0, 0.0, 0.0, 1.0);
>> }
>
> On the other hand, I was really happy to see that Paul did his tests
> this way.  I don't like the error tolerance stuff for driver debugging.
> I know, it makes sure that we're better than just 1/255 from the right
> answer, but when it comes to debugging all you get told is "pass" or
> "fail".  When you were debugging the matrix array access stuff, the bugs
> would have jumped out at us sooner to see a value smashed to 1.0 instead
> of approximately an expected color, or that just the zw channels were
> wrong.
>
> Maybe we need to just be doing these tests into floating point FBOs?
> Then we could do error tolerance in our probes instead of the draw, not
> be limited in ranges, and not flash windows on the screen in -auto mode.
> And we could do fancy presentation based on the probe expected values
> for non-"-auto" mode.  Wouldn't that be awesome?
>

I refactored the code a bit today so that it can produce two types of tests now:

A. Tests that apply a scale and offset and then use "probe rgba", so
they're only testing to one part in 255 (these are the tests I
originally produced, which Eric was voicing support for).

B. Tests that compare with an expected value (and tolerance) in GLSL,
and produce a solid red or green output to indicate pass or fail
(Ian's idea).

I'll send the diff to the list once I've got it a little more cleaned
up, probably on Monday.

On the one hand, automatically computing the correct tolerances was
tricky, and I'm still not certain I got it right (I still need to
think more).  Also, my experience so far confirms Eric's suspicion
that test set A is easier to debug with.

On the other hand, test set B uncovered some Mesa bugs that test set A
failed to find.  Specifically, Mesa's formulas for radians(), asin(),
acos(), and atan() are inaccurate, some of them embarrassingly so.
asin(), for example, has a worst-case absolute error of 0.45%, and its
worst case relative error is unbounded (asin(x) approaches +/- 6.75e-5
as x approaches 0).  But that's a topic for the Mesa mailing list I
suppose.

Personally, I think the fact that test set B found more bugs is a huge
feather in its cap, and probably trumps the debuggability issue for
now.  But I do like Eric's floating point FBO idea as a future
direction to take things in.


More information about the Piglit mailing list