[Piglit] [PATCH 0/15] Add support for shader_runner to run more than one test per process

[Nicolai Hähnle]

Thanks for doing that, the numbers are impressive.

Two small comments on the C parts; it's been too long since I looked 
into the Python parts to be able to properly comment.

Cheers,
Nicolai

On 09.09.2016 21:18, Dylan Baker wrote:
> This allows shader_runner to consume more than one test at a time. When
> it does this it reports the results as subtests rather than as regular
> tests. It also updates the python framework to handle and resume from
> crashes in the tests. When this mode is enabled, shader_runner runs an
> entire directory of shader_test files at a time.
>
> This makes shader_runner (which takes up the bulk of piglit) run much
> faster, even compared to per-process with fast skipping enabled, but at
> the cost of the robustness and precision of process isolation. For that
> reason its an option that must be enabled, the default should always be
> robustness not speed. This can be forced on in piglit.conf.
>
> From a performance standpoint it's impressive (this is a 4 thread HSW):
> PIGLIT_NO_FAST_SKIP=1 ./piglit run shader ns_w_is -c
>     936.35s user 611.18s system 301% cpu 8:33.71 total
> ./piglit run shader fs_w_is -c
>     726.21s user 398.25s system 306% cpu 6:07.23 total
> PIGLIT_NO_FAST_SKIP=1 ./piglit run shader ns_no_is -c --no-process-isolation
>     334.21s user 122.04s system 201% cpu 3:46.55 total
> ./piglit run shader fs_no_is -c --no-process-isolation
>     160.47s user 7.83s system 293% cpu 57.327 total
>
> For the general use case (With fast skipping) this is a reduction of 5
> minutes of runtime. With fast-skipping seems pretty useful, as it
> reduces runtime by 3 minutes even with the removal of process isolation.
>
> Dylan Baker (9):
>   framework/results: Lower subtest names
>   framework: Split the file parsing code out of ShaderTest
>   framework: Split FastSkipMixin.
>   framework: Bump JSON on disk format to version 9
>   framework: Add a Mixin class for running multiple tests in a single process
>   squash! shader_runner: add ability to run multiple tests per process
>   framework: Add class for running multiple shader_tests in a single process
>   framework: Plug in fast-skipping to MultiShaderTest
>   framework: add boilerplate to turn off process isolation
>
> Marek Olšák (6):
>   shader_runner: propagate all errors to piglit_init
>   util: guard against double context destruction
>   util: always set piglit_is_core_profile
>   util: rename piglit_gl_reinitialize_extensions->piglit_gl_invalidate_extensions
>   util: move PIGLIT_PATH_SEP to piglit-util.h
>   shader_runner: add ability to run multiple tests per process
>
>  framework/backends/json.py                             |  18 +-
>  framework/options.py                                   |   1 +-
>  framework/programs/run.py                              |  22 +-
>  framework/results.py                                   |   8 +-
>  framework/test/base.py                                 | 155 ++-
>  framework/test/deqp.py                                 |   4 +-
>  framework/test/opengl.py                               | 182 ++-
>  framework/test/shader_test.py                          | 201 +++-
>  piglit.conf.example                                    |   7 +-
>  tests/all.py                                           |  34 +-
>  tests/shader.py                                        |   4 +-
>  tests/shaders/shader_runner.c                          | 560 +++++++---
>  tests/util/piglit-framework-gl.c                       |   4 +-
>  tests/util/piglit-framework-gl/piglit_glut_framework.c |   7 +-
>  tests/util/piglit-framework-gl/piglit_wfl_framework.c  |   5 +-
>  tests/util/piglit-util-gl.c                            |   2 +-
>  tests/util/piglit-util-gl.h                            |   2 +-
>  tests/util/piglit-util.c                               |   6 +-
>  tests/util/piglit-util.h                               |   6 +-
>  unittests/framework/backends/schema/piglit-9.json      | 122 ++-
>  unittests/framework/backends/shared.py                 |   4 +-
>  unittests/framework/backends/test_json_update.py       |  85 ++-
>  unittests/framework/test/test_base.py                  | 138 +-
>  unittests/framework/test/test_opengl.py                | 192 +--
>  unittests/framework/test/test_shader_test.py           |  63 +-
>  25 files changed, 1514 insertions(+), 318 deletions(-)
>  create mode 100644 unittests/framework/backends/schema/piglit-9.json
>
>
>
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/piglit
>