[Piglit] Suggestions for creating statistical information from piglit results

[Ilia Mirkin]

On Wed, May 14, 2014 at 3:11 AM, Timothy Arceri <t_arceri at yahoo.com.au> wrote:
> After Valve's Rich Geldreich blasting of OpenGL drivers and reading this
> post [2] which trys to compare the driver quality of different drivers
> using the g-truc samples [3]
> I'm now interested in creating a similar graph and html page with a
> breakdown of pass/fail into gl version categories similar to the pdf
> provided on the website. This would provide some good stats on the real
> quality of the drivers given piglits wide range of tests.
>
> My question to those more knowledgeable about the inner working of
> piglit is what is the easiest way to create this? Obviously I need to
> add some functionality to the summary creation tool. But I also need a
> way to categories the tests into there respective gl version, glsl
> version and/or extension something thats not currently output to the
> results file. How would you suggest adding/extracting that information?
> I guess adding three new fields to the results might be useful:
> "gl-version":
> "glsl-vesion":
> "extension"
>
> But I'm not sure how I should extract that information and where in the
> code this should be implemented? Is this even possible? I assume I would
> at least have to have a table on the summary generation side to be able
> to map the test to the correct category as if say the require version
> for the test was used to work out the gl-version that wouldn't
> automatically mean that extension was part of that gl version.
>
> Anyway let me know your thoughts.

You can look at the test names reported by piglit. Almost all tests
follow a reasonable convention. For example test names might be

spec/!OpenGL 1.1/depthstencil-default_fb-blit samples=2
spec/ARB_depth_texture/fbo-generatemipmap-formats/GL_DEPTH_COMPONENT
spec/glsl-1.50/execution/fragcoord-layout-qualifiers-conflicting-case-1

With just a handful of rules, you could classify the majority of
tests. A bunch will be tricky, like the glean tests, I just wouldn't
worry about them. Of course some extensions are also part of some gl
versions, so you'd need a mapping of those. You can take a look at the
one I created in
http://people.freedesktop.org/~imirkin/glxinfo/glxinfo.js for an
unrelated reason.

Unrelatedly, note that e.g. the NVIDIA blob fails a *ton* of tests for
reasons completely unrelated to the test. E.g. they don't like the
#version in combination with something, their parser for that version
doesn't work properly, etc.

  -ilia