[Piglit] [PATCH 0/5] Randomized UBO tests of doom

[Dylan Baker]

Here's my idea.

We add ubo-fuzzer as a profile. We generate each test file on demand, and
write it out to a temporary file before running it. Then if the test fails
we can record the test into the results file for the developer to come back
and look at later to decide if it's worth keeping. Does this sound like a
reasonable approach?

On Mon, Nov 9, 2015 at 5:08 PM, Ian Romanick <idr at freedesktop.org> wrote:

> On 11/09/2015 05:19 AM, Ilia Mirkin wrote:
> > On Thu, Sep 25, 2014 at 4:39 PM, Ian Romanick <idr at freedesktop.org>
> wrote:
> >> On 09/24/2014 09:47 AM, Ian Romanick wrote:
> >>> So, here it is.  Finally.
> >>>
> >>> The first two patches provide the infrastructure for generating
> >>> randomized UBO tests.  I think these are pretty solid, but there are
> >>> probably ways to impove the Python, etc.
> >>>
> >>> The remaining three patches are examples of ways the infrastructure can
> >>> be used.  Here is where I am not sure what we should do.  I know that
> we
> >>> don't want to make the "forever" test in patch 4 part of regular piglit
> >>> runs.  However, it has found a LOT of bugs in EVERY OpenGL driver that
> I
> >>> have tested.
> >>>
> >>> I'm also unsure about the random tests generated by patch 3.  Do we
> want
> >>> actual random tests in regular piglit runs?  What do we do for tests
> for
> >>> GLSL 1.40?  Generate the "same" tests, but use #version 140 instead of
> >>> #extension?
> >>>
> >>> In any case, I know that folks are hard at work on fp64 support, so
> >>> using the various random runners here should help that effort.  Sorry
> >>> for all the delays.
> >>>
> >>> One last thing... I'm presenting a bunch of information about this work
> >>> at XDC in a couple weeks.  Maybe we want to wait to hammer out the more
> >>> difficult details until then.  Dunno.
> >>
> >> I've pushed updated version of this series to the ubo-lolz branch of my
> >> fd.o piglit repo.
> >
> > It looks like this didn't end up going anywhere... on several
> > occasions I've either used this script (like for fp64), or recommended
> > it to others (like for ssbo, and will do so for ARB_enhanced_layouts
> > when that conversation comes up).
> >
> > I think it'd benefit greatly from being in a shared and updated
> > location as features are added, bugs are fixed, etc. However running
> > it as part of piglit may not be a great idea. Perhaps we can find a
> > place in the repo where we can store it? Or maybe even a different
> > repo?
> >
> > How about tests/fuzzing in piglit? Any objections?
>
> Having it actually live somewhere is a good idea.  There are definitely
> some bugs in it... and some of the tests that failed on other
> implementations may be expecting things the spec doesn't allow.  I need
> to dig back through my e-mail, but some guys from NVIDIA had convinced
> me that there was something wrong... but I don't recall what.
>
> When I presented this at XDC in 2014, I think the consensus was that
> tests that actually found a bug should be added to the "right" place in
> the repo, but we don't want to run 47,000,000 random tests on regular
> piglit runs.  Someone should have a system somewhere that just runs
> these (and other) random, fuzzing tests 24/7.
>
> > Cheers,
> >
> >   -ilia
>
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/piglit
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/piglit/attachments/20151109/767c5ca9/attachment.html>