[Piglit] [PATCH 0/11] Add support for running multiple tests in a single process

[Daniel Vetter]

On Fri, Aug 26, 2016 at 11:31:06AM -0700, Dylan Baker wrote:
> Piglit has always assumed that a 1:1 relationship between test instances
> and processes. This is very unique to piglit among test suites, and has
> some real advantages for testing hardware drivers. For one thing it
> means that one test doesn't change another test's status by polluting
> the OpenGL state, for another it makes handling crashes very simple
> (since the python runner is a separate process from the tests). It does
> have a major drawback, and that's the cost of starting processes,
> especially if the tests are badly designed. While the former affects all
> piglit tests, the second problem has especially noticeable effects on
> the Vulkan CTS (tests/deqp_vk.py).
> 
> This series sets about solving two related problems:
> First, it builds the necessary groundwork for creating Test instances
> (the process) that have multiple tests in them. There are several
> potential issues that it needs to address:
>  - Getting the results out
>  - Making sure that all of the tests ran (and no additional tests ran)
>  - Detecting premature exit of the test process
>  - Resuming the remaining tests.
> 
> This mixin directly solves the last three issues, though it provides
> some interfaces that subclasses may need to overwrite to work correctly.
> Getting the results out however, requires writing a custom
> interpret_result method that is aware of what the mixin does to put the
> output of the tests together.
> 
> This was solved in a generic way because of another bit of work that
> Marek started and I'm trying to get finished, which is doing much the
> same thing to shader runner.
> 
> The second problem it solves is implementing a dEQP wrapper class that
> can run multiple tests in a single process. This includes a lot of
> boilerplate code to build up the profile with either the existing 1:1
> (process:tests) style testing or the new 1:M style tests.
> 
> These are implemented for deqp_vk (which was the primary motivation for
> this series), and the cts_* tests, but not for the deqp_* tests. This is
> because there is no performance incentive with the deqp_* tests and it
> has the disadvantage of removing the process isolation.
> 
> For deqp_vk I saw an 8x speedup moving from single mode to group mode on an 8
> thread machines. On the same machine the OpenGL and OpenGL ES CTSes showed
> roughly on par performance.
> 
> Dylan Baker (11):
>   framework/test/deqp: Fix pylint warnings.
>   unittests: mark deqp_egl integration test as xfail
>   framework: Bump JSON on disk format to version 9
>   framework: Add a Mixin class for running multiple tests in a single process
>   framework: Add an initial implementaiton of a DEQPGroupTest
>   framework: Implement boilerplate to generate a group-at-a-time profile
>   tests/deqp_vk: add support for group at a time mode
>   tests/cts_gl: Add support for group-at-a-time mode
>   tests/cts_gles: Add support for group-at-a-time mode
>   tests: Add support for deqp_gles* for group at a time mode
>   tests/deqp_egl: add group at a time mode support.

Hm, this could be very interesting for igt too. Lots of testcases have
some costly setup, and running them in one go dramatically amortizes that.
At least in some tests. But then there's the downside that sometimes tests
leak between different subtests and eventually cause failures. Not sure
I'll have time to play around with this anytime soon unfortunately.
-Daniel

> 
>  framework/backends/json.py                        |  18 +-
>  framework/options.py                              |   3 +-
>  framework/programs/run.py                         |  10 +-
>  framework/results.py                              |   2 +-
>  framework/test/base.py                            | 134 ++++-
>  framework/test/deqp.py                            | 331 +++++++++++--
>  tests/cts_gl.py                                   |  22 +-
>  tests/cts_gles.py                                 |  22 +-
>  tests/deqp_egl.py                                 |  14 +-
>  tests/deqp_gles2.py                               |   7 +-
>  tests/deqp_gles3.py                               |   7 +-
>  tests/deqp_gles31.py                              |   7 +-
>  tests/deqp_vk.py                                  |  36 +-
>  tox.ini                                           |   1 +-
>  unittests/framework/backends/schema/piglit-9.json | 122 +++++-
>  unittests/framework/backends/shared.py            |   4 +-
>  unittests/framework/backends/test_json_update.py  |  85 +++-
>  unittests/framework/test/test_base.py             | 138 ++++-
>  unittests/framework/test/test_deqp.py             | 395 ++++++++++++---
>  unittests/suites/test_integration.py              |   3 +-
>  20 files changed, 1221 insertions(+), 140 deletions(-)
>  create mode 100644 unittests/framework/backends/schema/piglit-9.json
> 
> -- 
> git-series 0.8.7
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/piglit

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch