[Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Ilia Mirkin]

That's a full step ahead of my suggestion. I just want the thing in a
shared place so that I can run it when doing dodgy things to code I
don't understand... if there's some automated process running it,
that's just gravy on top.

On Tue, Nov 10, 2015 at 1:26 AM, Dylan Baker <baker.dylan.c at gmail.com> wrote:
> Here's my idea.
>
> We add ubo-fuzzer as a profile. We generate each test file on demand, and
> write it out to a temporary file before running it. Then if the test fails
> we can record the test into the results file for the developer to come back
> and look at later to decide if it's worth keeping. Does this sound like a
> reasonable approach?
>
> On Mon, Nov 9, 2015 at 5:08 PM, Ian Romanick <idr at freedesktop.org> wrote:
>>
>> On 11/09/2015 05:19 AM, Ilia Mirkin wrote:
>> > On Thu, Sep 25, 2014 at 4:39 PM, Ian Romanick <idr at freedesktop.org>
>> > wrote:
>> >> On 09/24/2014 09:47 AM, Ian Romanick wrote:
>> >>> So, here it is.  Finally.
>> >>>
>> >>> The first two patches provide the infrastructure for generating
>> >>> randomized UBO tests.  I think these are pretty solid, but there are
>> >>> probably ways to impove the Python, etc.
>> >>>
>> >>> The remaining three patches are examples of ways the infrastructure
>> >>> can
>> >>> be used.  Here is where I am not sure what we should do.  I know that
>> >>> we
>> >>> don't want to make the "forever" test in patch 4 part of regular
>> >>> piglit
>> >>> runs.  However, it has found a LOT of bugs in EVERY OpenGL driver that
>> >>> I
>> >>> have tested.
>> >>>
>> >>> I'm also unsure about the random tests generated by patch 3.  Do we
>> >>> want
>> >>> actual random tests in regular piglit runs?  What do we do for tests
>> >>> for
>> >>> GLSL 1.40?  Generate the "same" tests, but use #version 140 instead of
>> >>> #extension?
>> >>>
>> >>> In any case, I know that folks are hard at work on fp64 support, so
>> >>> using the various random runners here should help that effort.  Sorry
>> >>> for all the delays.
>> >>>
>> >>> One last thing... I'm presenting a bunch of information about this
>> >>> work
>> >>> at XDC in a couple weeks.  Maybe we want to wait to hammer out the
>> >>> more
>> >>> difficult details until then.  Dunno.
>> >>
>> >> I've pushed updated version of this series to the ubo-lolz branch of my
>> >> fd.o piglit repo.
>> >
>> > It looks like this didn't end up going anywhere... on several
>> > occasions I've either used this script (like for fp64), or recommended
>> > it to others (like for ssbo, and will do so for ARB_enhanced_layouts
>> > when that conversation comes up).
>> >
>> > I think it'd benefit greatly from being in a shared and updated
>> > location as features are added, bugs are fixed, etc. However running
>> > it as part of piglit may not be a great idea. Perhaps we can find a
>> > place in the repo where we can store it? Or maybe even a different
>> > repo?
>> >
>> > How about tests/fuzzing in piglit? Any objections?
>>
>> Having it actually live somewhere is a good idea.  There are definitely
>> some bugs in it... and some of the tests that failed on other
>> implementations may be expecting things the spec doesn't allow.  I need
>> to dig back through my e-mail, but some guys from NVIDIA had convinced
>> me that there was something wrong... but I don't recall what.
>>
>> When I presented this at XDC in 2014, I think the consensus was that
>> tests that actually found a bug should be added to the "right" place in
>> the repo, but we don't want to run 47,000,000 random tests on regular
>> piglit runs.  Someone should have a system somewhere that just runs
>> these (and other) random, fuzzing tests 24/7.
>>
>> > Cheers,
>> >
>> >   -ilia
>>
>> _______________________________________________
>> Piglit mailing list
>> Piglit at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/piglit
>
>