[Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Ian Romanick]

On 09/24/2014 10:24 AM, Matt Turner wrote:
> On Wed, Sep 24, 2014 at 9:47 AM, Ian Romanick <idr at freedesktop.org> wrote:
>> So, here it is.  Finally.
>>
>> The first two patches provide the infrastructure for generating
>> randomized UBO tests.  I think these are pretty solid, but there are
>> probably ways to impove the Python, etc.
>>
>> The remaining three patches are examples of ways the infrastructure can
>> be used.  Here is where I am not sure what we should do.  I know that we
>> don't want to make the "forever" test in patch 4 part of regular piglit
>> runs.  However, it has found a LOT of bugs in EVERY OpenGL driver that I
>> have tested.
>>
>> I'm also unsure about the random tests generated by patch 3.  Do we want
>> actual random tests in regular piglit runs?
> 
> I think you know the answer to that. :)

I actually don't. :( Running the same set of tests repeatedly isn't
going to find additional bugs.  If we're at a point where we think we've
fixed "all" the bugs, running slightly different
array-of-structures-containing-a-structure test won't have the
irritating oscillating behavior that, say, the front-buffer rendering
tests or timer query tests have.  The tests are generated at build-time,
so the results will be stable across runs regardless.

Are there more bugs to be found by this generator in our implementation?
 I don't know.  The probability diminishes as more things get fixed in
the implementation.  As we add more functionality, the probability goes
back up.

What method should we use to search for them?  I also don't know.  Last
week I ran the script in patch 4 for about 8 hours on a single CPU
without encountering a failure.  It ran around 8,000 tests.

This is part of the reason that it has taken so long to get this posted.
 That and fixing the bugs that it found...

>> What do we do for tests for
>> GLSL 1.40?  Generate the "same" tests, but use #version 140 instead of
>> #extension?
> 
> I don't see any benefit to running the same execution tests under
> different GLSL versions.

Certainly not the same.  Almost certainly not "fuzzy" the same either.
We usually have tests for an extension and tests for the core version
(with varying degrees of success).  It looks like the only
implementation (possibly existing in the world) that has UBOs and not
OpenGL 3.1 is swrast.  If we don't care about abusing swrast, we could
just use these tests on 3.1.