[Mesa-dev] [RFC shader-db] Add support for shadertoy tests

[Erik Faye-Lund]

On Wed, Jun 24, 2015 at 2:18 AM, Rob Clark <robdclark at gmail.com> wrote:
> On Tue, Jun 23, 2015 at 7:27 PM, Dylan Baker <baker.dylan.c at gmail.com> wrote:
>> I have a couple of python pointers for you, feel free to take them or
>> leave them.
>
> cool, thanks..
>
> What do others think about including shadertoy in shader-db?  If it is
> a useful thing, I'll clean up my script and re-submit..

It's a pretty neat resource, but AFAIK it has a few (probably minor)
potential problems:

1) The data-set changes over time. This might not be a real-world
   problem, as measurements are presumably taken with two identical
   snap-shots.
2) It contains a *lot* of basically the same thing; raymarched
   raytracers. So I worry a bit that pulling in all of the
   Shadertoy-shaders ends up punishing "normal" shaders, by making them
   seem more fringe. The raymarchers on Shadertoy tends to be extremely
   ALU heavy compared to games shaders, and do very long loops. I have no
   doubt it's healthy to have *some* such shaders in shader-db, but
   perhaps all is overkill?
3) Since all use the same vertex shader, won't changes to that get
   artificially high scores?

I'm not saying these are showstoppers, just something that might need
addressing.