[Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Ian Romanick]

On 11/09/2015 05:19 AM, Ilia Mirkin wrote:
> On Thu, Sep 25, 2014 at 4:39 PM, Ian Romanick <idr at freedesktop.org> wrote:
>> On 09/24/2014 09:47 AM, Ian Romanick wrote:
>>> So, here it is.  Finally.
>>>
>>> The first two patches provide the infrastructure for generating
>>> randomized UBO tests.  I think these are pretty solid, but there are
>>> probably ways to impove the Python, etc.
>>>
>>> The remaining three patches are examples of ways the infrastructure can
>>> be used.  Here is where I am not sure what we should do.  I know that we
>>> don't want to make the "forever" test in patch 4 part of regular piglit
>>> runs.  However, it has found a LOT of bugs in EVERY OpenGL driver that I
>>> have tested.
>>>
>>> I'm also unsure about the random tests generated by patch 3.  Do we want
>>> actual random tests in regular piglit runs?  What do we do for tests for
>>> GLSL 1.40?  Generate the "same" tests, but use #version 140 instead of
>>> #extension?
>>>
>>> In any case, I know that folks are hard at work on fp64 support, so
>>> using the various random runners here should help that effort.  Sorry
>>> for all the delays.
>>>
>>> One last thing... I'm presenting a bunch of information about this work
>>> at XDC in a couple weeks.  Maybe we want to wait to hammer out the more
>>> difficult details until then.  Dunno.
>>
>> I've pushed updated version of this series to the ubo-lolz branch of my
>> fd.o piglit repo.
> 
> It looks like this didn't end up going anywhere... on several
> occasions I've either used this script (like for fp64), or recommended
> it to others (like for ssbo, and will do so for ARB_enhanced_layouts
> when that conversation comes up).
> 
> I think it'd benefit greatly from being in a shared and updated
> location as features are added, bugs are fixed, etc. However running
> it as part of piglit may not be a great idea. Perhaps we can find a
> place in the repo where we can store it? Or maybe even a different
> repo?
> 
> How about tests/fuzzing in piglit? Any objections?

Having it actually live somewhere is a good idea.  There are definitely
some bugs in it... and some of the tests that failed on other
implementations may be expecting things the spec doesn't allow.  I need
to dig back through my e-mail, but some guys from NVIDIA had convinced
me that there was something wrong... but I don't recall what.

When I presented this at XDC in 2014, I think the consensus was that
tests that actually found a bug should be added to the "right" place in
the repo, but we don't want to run 47,000,000 random tests on regular
piglit runs.  Someone should have a system somewhere that just runs
these (and other) random, fuzzing tests 24/7.

> Cheers,
> 
>   -ilia