[Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Arthur Huillet]

On 2015-11-10 2:08, Ian Romanick wrote:
> On 11/09/2015 05:19 AM, Ilia Mirkin wrote:
>> On Thu, Sep 25, 2014 at 4:39 PM, Ian Romanick <idr at freedesktop.org> 
>> wrote:
>>> On 09/24/2014 09:47 AM, Ian Romanick wrote:
>>>> So, here it is.  Finally.
>>>> 
>>>> The first two patches provide the infrastructure for generating
>>>> randomized UBO tests.  I think these are pretty solid, but there are
>>>> probably ways to impove the Python, etc.
>>>> 
>>>> The remaining three patches are examples of ways the infrastructure 
>>>> can
>>>> be used.  Here is where I am not sure what we should do.  I know 
>>>> that we
>>>> don't want to make the "forever" test in patch 4 part of regular 
>>>> piglit
>>>> runs.  However, it has found a LOT of bugs in EVERY OpenGL driver 
>>>> that I
>>>> have tested.
>>>> 
>>>> I'm also unsure about the random tests generated by patch 3.  Do we 
>>>> want
>>>> actual random tests in regular piglit runs?  What do we do for tests 
>>>> for
>>>> GLSL 1.40?  Generate the "same" tests, but use #version 140 instead 
>>>> of
>>>> #extension?
>>>> 
>>>> In any case, I know that folks are hard at work on fp64 support, so
>>>> using the various random runners here should help that effort.  
>>>> Sorry
>>>> for all the delays.
>>>> 
>>>> One last thing... I'm presenting a bunch of information about this 
>>>> work
>>>> at XDC in a couple weeks.  Maybe we want to wait to hammer out the 
>>>> more
>>>> difficult details until then.  Dunno.
>>> 
>>> I've pushed updated version of this series to the ubo-lolz branch of 
>>> my
>>> fd.o piglit repo.
>> 
>> It looks like this didn't end up going anywhere... on several
>> occasions I've either used this script (like for fp64), or recommended
>> it to others (like for ssbo, and will do so for ARB_enhanced_layouts
>> when that conversation comes up).
>> 
>> I think it'd benefit greatly from being in a shared and updated
>> location as features are added, bugs are fixed, etc. However running
>> it as part of piglit may not be a great idea. Perhaps we can find a
>> place in the repo where we can store it? Or maybe even a different
>> repo?
>> 
>> How about tests/fuzzing in piglit? Any objections?
> 
> Having it actually live somewhere is a good idea.  There are definitely
> some bugs in it... and some of the tests that failed on other
> implementations may be expecting things the spec doesn't allow.  I need
> to dig back through my e-mail, but some guys from NVIDIA had convinced
> me that there was something wrong... but I don't recall what.

I am "some guys" (or at least one of them). IIRC, the bug was in 
shader-runner, not in your fuzzer per se, and got fixed in 
http://cgit.freedesktop.org/piglit/commit/?id=2b94faec18dc1f8f0d9241ec731408959320cd7c

I don't recall finding other bugs, there was one in the NVIDIA driver 
that it exposed and we fixed it. (Thanks!)

> When I presented this at XDC in 2014, I think the consensus was that
> tests that actually found a bug should be added to the "right" place in
> the repo, but we don't want to run 47,000,000 random tests on regular
> piglit runs.

I think it makes sense to only store the tests that are, or used to be, 
red, and not store the tests that always passed everywhere, as these 
bring no particular value.

-- 
Greetings,
A. Huillet
NVIDIA Linux graphics