[Piglit] [PATCH 0/11] Add support for running multiple tests in a single process

[Dylan Baker]

Piglit has always assumed that a 1:1 relationship between test instances
and processes. This is very unique to piglit among test suites, and has
some real advantages for testing hardware drivers. For one thing it
means that one test doesn't change another test's status by polluting
the OpenGL state, for another it makes handling crashes very simple
(since the python runner is a separate process from the tests). It does
have a major drawback, and that's the cost of starting processes,
especially if the tests are badly designed. While the former affects all
piglit tests, the second problem has especially noticeable effects on
the Vulkan CTS (tests/deqp_vk.py).

This series sets about solving two related problems:
First, it builds the necessary groundwork for creating Test instances
(the process) that have multiple tests in them. There are several
potential issues that it needs to address:
 - Getting the results out
 - Making sure that all of the tests ran (and no additional tests ran)
 - Detecting premature exit of the test process
 - Resuming the remaining tests.

This mixin directly solves the last three issues, though it provides
some interfaces that subclasses may need to overwrite to work correctly.
Getting the results out however, requires writing a custom
interpret_result method that is aware of what the mixin does to put the
output of the tests together.

This was solved in a generic way because of another bit of work that
Marek started and I'm trying to get finished, which is doing much the
same thing to shader runner.

The second problem it solves is implementing a dEQP wrapper class that
can run multiple tests in a single process. This includes a lot of
boilerplate code to build up the profile with either the existing 1:1
(process:tests) style testing or the new 1:M style tests.

These are implemented for deqp_vk (which was the primary motivation for
this series), and the cts_* tests, but not for the deqp_* tests. This is
because there is no performance incentive with the deqp_* tests and it
has the disadvantage of removing the process isolation.

For deqp_vk I saw an 8x speedup moving from single mode to group mode on an 8
thread machines. On the same machine the OpenGL and OpenGL ES CTSes showed
roughly on par performance.

Dylan Baker (11):
  framework/test/deqp: Fix pylint warnings.
  unittests: mark deqp_egl integration test as xfail
  framework: Bump JSON on disk format to version 9
  framework: Add a Mixin class for running multiple tests in a single process
  framework: Add an initial implementaiton of a DEQPGroupTest
  framework: Implement boilerplate to generate a group-at-a-time profile
  tests/deqp_vk: add support for group at a time mode
  tests/cts_gl: Add support for group-at-a-time mode
  tests/cts_gles: Add support for group-at-a-time mode
  tests: Add support for deqp_gles* for group at a time mode
  tests/deqp_egl: add group at a time mode support.

 framework/backends/json.py                        |  18 +-
 framework/options.py                              |   3 +-
 framework/programs/run.py                         |  10 +-
 framework/results.py                              |   2 +-
 framework/test/base.py                            | 134 ++++-
 framework/test/deqp.py                            | 331 +++++++++++--
 tests/cts_gl.py                                   |  22 +-
 tests/cts_gles.py                                 |  22 +-
 tests/deqp_egl.py                                 |  14 +-
 tests/deqp_gles2.py                               |   7 +-
 tests/deqp_gles3.py                               |   7 +-
 tests/deqp_gles31.py                              |   7 +-
 tests/deqp_vk.py                                  |  36 +-
 tox.ini                                           |   1 +-
 unittests/framework/backends/schema/piglit-9.json | 122 +++++-
 unittests/framework/backends/shared.py            |   4 +-
 unittests/framework/backends/test_json_update.py  |  85 +++-
 unittests/framework/test/test_base.py             | 138 ++++-
 unittests/framework/test/test_deqp.py             | 395 ++++++++++++---
 unittests/suites/test_integration.py              |   3 +-
 20 files changed, 1221 insertions(+), 140 deletions(-)
 create mode 100644 unittests/framework/backends/schema/piglit-9.json

-- 
git-series 0.8.7