[Piglit] [PATCH 0/4] RFC: apitrace test runner in piglit

[Eric Anholt]

Here's a little project I started a couple of weeks ago.  I think
piglit is great for having very debuggable, targeted testcases of GL,
but we're weak on testing complicated shaders or significant
texturing.  So, what if we just took traces of a bunch of
applications, visually inspected them on our drivers compared to other
drivers, and made sure that they kept looking good in the future?
This would make me a lot more confident in things like tricky discard
optimizations or loop control flow or instruction scheduling.

This series does not create a directory of tests in piglit, nor do I
plan to.  I was thinking we could handle it like shader-db: take a
bunch of traces of open source apps and put them in a separate repo.
You download that shared repo, add your own traces of non-open-source
stuff that you care about, inspect and save your golden images for
your driver, and link to that from piglit.

The tricky part of distributing the traces is going to be storage.
Traces are big, you probably don't want to download the history of the
repo, and you may not even want to download the full repo.  Thoughts I've had:

1) Only small traces in a shared repo (that way it could be on fd.o).
2) git-annex tons of stuff and put it up on S3 or whatever
3) git-lfs tons of stuff on github.

I don't really like any of these solutions.  Still, I think people
might find this useful even if I don't solve the shared traces
problem.  So, what do you think?

Eric Anholt (4):
  README: Add a note of how to run the framework tests
  framework: Add a driver classifier based on the renderer string.
  apitrace: Add a script for capturing and comparing trace images.
  all.py: Test any apitraces we find under tests/apitrace/traces.

 README                                        |   2 +-
 framework/driver_classifier.py                | 126 ++++++++++++++++++++
 framework/test/piglit_test.py                 |   1 +
 tests/all.py                                  |  22 +++-
 tests/apitrace/test-trace.py                  | 162 ++++++++++++++++++++++++++
 unittests/framework/test_driver_classifier.py |  59 ++++++++++
 6 files changed, 370 insertions(+), 2 deletions(-)
 create mode 100644 framework/driver_classifier.py
 create mode 100755 tests/apitrace/test-trace.py
 create mode 100644 unittests/framework/test_driver_classifier.py

-- 
2.9.3