[Piglit] [PATCH v2 0/16] Add support for shader_runner to run more than one test per process

[Dylan Baker]

This allows shader_runner to consume more than one test at a time. When
it does this it reports the results as subtests rather than as regular
tests. It also updates the python framework to handle and resume from
crashes in the tests. When this mode is enabled, shader_runner runs an
entire directory of shader_test files at a time.

This makes shader_runner (which takes up the bulk of piglit) run much
faster, even compared to per-process with fast skipping enabled, but at
the cost of the robustness and reproducibility of process isolation. For that
reason its an option that must be enabled, the default should always be
robustness not speed. This can be turned on permanently on in piglit.conf.

>From a performance standpoint it's impressive (this is a 4 thread HSW):
PIGLIT_NO_FAST_SKIP=1 ./piglit run shader ns_w_is -c
    936.35s user 611.18s system 301% cpu 8:33.71 total
./piglit run shader fs_w_is -c
    726.21s user 398.25s system 306% cpu 6:07.23 total
PIGLIT_NO_FAST_SKIP=1 ./piglit run shader ns_no_is -c --process-isolation=false
    334.21s user 122.04s system 201% cpu 3:46.55 total
./piglit run shader fs_no_is -c --process-isolation=false
    160.47s user 7.83s system 293% cpu 57.327 total

For the general use case (With fast skipping) this is a reduction of 5
minutes of runtime.

This work is part mine, and partially from Marek.

Changes since v1:
 - Add the second squash! commit, which is all v1 changes

The biggest changes are fixing more GL vs GLES differences, doing all of
our guarding at the front of the function, and doing a better job
cleaning up between tests.

Dylan Baker (10):
  framework/results: Lower subtest names
  framework: Split the file parsing code out of ShaderTest
  framework: Split FastSkipMixin.
  framework: Bump JSON on disk format to version 9
  framework: Add a Mixin class for running multiple tests in a single process
  squash! shader_runner: add ability to run multiple tests per process
  squash! shader_runner: add ability to run multiple tests per process
  framework: Add class for running multiple shader_tests in a single process
  framework: Plug in fast-skipping to MultiShaderTest
  framework: add boilerplate to turn off process isolation

Marek Olšák (6):
  shader_runner: propagate all errors to piglit_init
  util: guard against double context destruction
  util: always set piglit_is_core_profile
  util: rename piglit_gl_reinitialize_extensions->piglit_gl_invalidate_extensions
  util: move PIGLIT_PATH_SEP to piglit-util.h
  shader_runner: add ability to run multiple tests per process

 framework/backends/json.py                             |  18 +-
 framework/options.py                                   |   1 +-
 framework/programs/run.py                              |  23 +-
 framework/results.py                                   |   8 +-
 framework/test/base.py                                 | 155 ++-
 framework/test/deqp.py                                 |   4 +-
 framework/test/opengl.py                               | 182 ++-
 framework/test/shader_test.py                          | 201 ++-
 piglit.conf.example                                    |   7 +-
 tests/all.py                                           |  37 +-
 tests/shader.py                                        |   4 +-
 tests/shaders/shader_runner.c                          | 582 +++++++---
 tests/util/piglit-framework-gl.c                       |   4 +-
 tests/util/piglit-framework-gl/piglit_glut_framework.c |   7 +-
 tests/util/piglit-framework-gl/piglit_wfl_framework.c  |   5 +-
 tests/util/piglit-util-gl.c                            |   2 +-
 tests/util/piglit-util-gl.h                            |   2 +-
 tests/util/piglit-util.c                               |   6 +-
 tests/util/piglit-util.h                               |   6 +-
 unittests/framework/backends/schema/piglit-9.json      | 122 ++-
 unittests/framework/backends/shared.py                 |   4 +-
 unittests/framework/backends/test_json_update.py       |  85 +-
 unittests/framework/test/test_base.py                  | 138 +-
 unittests/framework/test/test_opengl.py                | 192 +--
 unittests/framework/test/test_shader_test.py           |  63 +-
 25 files changed, 1533 insertions(+), 325 deletions(-)
 create mode 100644 unittests/framework/backends/schema/piglit-9.json

-- 
git-series 0.8.10