[Piglit] [PATCH v2 0/16] Add support for shader_runner to run more than one test per process

[Marek Olšák]

I reviewed all the C code changes and they look good to me.

I'd like this series to land whether or not somebody reviews the python changes.

Marek

On Fri, Sep 30, 2016 at 11:17 PM, Dylan Baker <dylan at pnwbakers.com> wrote:
> This allows shader_runner to consume more than one test at a time. When
> it does this it reports the results as subtests rather than as regular
> tests. It also updates the python framework to handle and resume from
> crashes in the tests. When this mode is enabled, shader_runner runs an
> entire directory of shader_test files at a time.
>
> This makes shader_runner (which takes up the bulk of piglit) run much
> faster, even compared to per-process with fast skipping enabled, but at
> the cost of the robustness and reproducibility of process isolation. For that
> reason its an option that must be enabled, the default should always be
> robustness not speed. This can be turned on permanently on in piglit.conf.
>
> From a performance standpoint it's impressive (this is a 4 thread HSW):
> PIGLIT_NO_FAST_SKIP=1 ./piglit run shader ns_w_is -c
>     936.35s user 611.18s system 301% cpu 8:33.71 total
> ./piglit run shader fs_w_is -c
>     726.21s user 398.25s system 306% cpu 6:07.23 total
> PIGLIT_NO_FAST_SKIP=1 ./piglit run shader ns_no_is -c --process-isolation=false
>     334.21s user 122.04s system 201% cpu 3:46.55 total
> ./piglit run shader fs_no_is -c --process-isolation=false
>     160.47s user 7.83s system 293% cpu 57.327 total
>
> For the general use case (With fast skipping) this is a reduction of 5
> minutes of runtime.
>
> This work is part mine, and partially from Marek.
>
> Changes since v1:
>  - Add the second squash! commit, which is all v1 changes
>
> The biggest changes are fixing more GL vs GLES differences, doing all of
> our guarding at the front of the function, and doing a better job
> cleaning up between tests.
>
> Dylan Baker (10):
>   framework/results: Lower subtest names
>   framework: Split the file parsing code out of ShaderTest
>   framework: Split FastSkipMixin.
>   framework: Bump JSON on disk format to version 9
>   framework: Add a Mixin class for running multiple tests in a single process
>   squash! shader_runner: add ability to run multiple tests per process
>   squash! shader_runner: add ability to run multiple tests per process
>   framework: Add class for running multiple shader_tests in a single process
>   framework: Plug in fast-skipping to MultiShaderTest
>   framework: add boilerplate to turn off process isolation
>
> Marek Olšák (6):
>   shader_runner: propagate all errors to piglit_init
>   util: guard against double context destruction
>   util: always set piglit_is_core_profile
>   util: rename piglit_gl_reinitialize_extensions->piglit_gl_invalidate_extensions
>   util: move PIGLIT_PATH_SEP to piglit-util.h
>   shader_runner: add ability to run multiple tests per process
>
>  framework/backends/json.py                             |  18 +-
>  framework/options.py                                   |   1 +-
>  framework/programs/run.py                              |  23 +-
>  framework/results.py                                   |   8 +-
>  framework/test/base.py                                 | 155 ++-
>  framework/test/deqp.py                                 |   4 +-
>  framework/test/opengl.py                               | 182 ++-
>  framework/test/shader_test.py                          | 201 ++-
>  piglit.conf.example                                    |   7 +-
>  tests/all.py                                           |  37 +-
>  tests/shader.py                                        |   4 +-
>  tests/shaders/shader_runner.c                          | 582 +++++++---
>  tests/util/piglit-framework-gl.c                       |   4 +-
>  tests/util/piglit-framework-gl/piglit_glut_framework.c |   7 +-
>  tests/util/piglit-framework-gl/piglit_wfl_framework.c  |   5 +-
>  tests/util/piglit-util-gl.c                            |   2 +-
>  tests/util/piglit-util-gl.h                            |   2 +-
>  tests/util/piglit-util.c                               |   6 +-
>  tests/util/piglit-util.h                               |   6 +-
>  unittests/framework/backends/schema/piglit-9.json      | 122 ++-
>  unittests/framework/backends/shared.py                 |   4 +-
>  unittests/framework/backends/test_json_update.py       |  85 +-
>  unittests/framework/test/test_base.py                  | 138 +-
>  unittests/framework/test/test_opengl.py                | 192 +--
>  unittests/framework/test/test_shader_test.py           |  63 +-
>  25 files changed, 1533 insertions(+), 325 deletions(-)
>  create mode 100644 unittests/framework/backends/schema/piglit-9.json
>
> --
> git-series 0.8.10
>
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/piglit
>