[Piglit] [PATCH 0/15] Add support for shader_runner to run more than one test per process

[Dylan Baker]

On Sat, Sep 10, 2016, at 06:56, Marek Olšák wrote:
> Thanks a lot of doing this!!
> 
> BTW, --no-process-isolation is not a valid parameter.

Yeah, I decided to change the parameter after I ran the benchmarks, when
I was writing documentation and proper commit messages. I realized that
I'd promised a piglit.conf option to force this on, and that would
either require two switches, or just changing the switch to take a
parameter to get consistent behavior.

So instead it's '--process-isolation' with an argument that can be
roughly translated as True or False (1, 0, true, false, yes, no; with
any capitalization). I forgot to update the cover letter.

Dylan

> 
> Marek
> 
> On Fri, Sep 9, 2016 at 9:18 PM, Dylan Baker <dylan at pnwbakers.com> wrote:
> > This allows shader_runner to consume more than one test at a time. When
> > it does this it reports the results as subtests rather than as regular
> > tests. It also updates the python framework to handle and resume from
> > crashes in the tests. When this mode is enabled, shader_runner runs an
> > entire directory of shader_test files at a time.
> >
> > This makes shader_runner (which takes up the bulk of piglit) run much
> > faster, even compared to per-process with fast skipping enabled, but at
> > the cost of the robustness and precision of process isolation. For that
> > reason its an option that must be enabled, the default should always be
> > robustness not speed. This can be forced on in piglit.conf.
> >
> > From a performance standpoint it's impressive (this is a 4 thread HSW):
> > PIGLIT_NO_FAST_SKIP=1 ./piglit run shader ns_w_is -c
> >     936.35s user 611.18s system 301% cpu 8:33.71 total
> > ./piglit run shader fs_w_is -c
> >     726.21s user 398.25s system 306% cpu 6:07.23 total
> > PIGLIT_NO_FAST_SKIP=1 ./piglit run shader ns_no_is -c --no-process-isolation
> >     334.21s user 122.04s system 201% cpu 3:46.55 total
> > ./piglit run shader fs_no_is -c --no-process-isolation
> >     160.47s user 7.83s system 293% cpu 57.327 total
> >
> > For the general use case (With fast skipping) this is a reduction of 5
> > minutes of runtime. With fast-skipping seems pretty useful, as it
> > reduces runtime by 3 minutes even with the removal of process isolation.
> >
> > Dylan Baker (9):
> >   framework/results: Lower subtest names
> >   framework: Split the file parsing code out of ShaderTest
> >   framework: Split FastSkipMixin.
> >   framework: Bump JSON on disk format to version 9
> >   framework: Add a Mixin class for running multiple tests in a single process
> >   squash! shader_runner: add ability to run multiple tests per process
> >   framework: Add class for running multiple shader_tests in a single process
> >   framework: Plug in fast-skipping to MultiShaderTest
> >   framework: add boilerplate to turn off process isolation
> >
> > Marek Olšák (6):
> >   shader_runner: propagate all errors to piglit_init
> >   util: guard against double context destruction
> >   util: always set piglit_is_core_profile
> >   util: rename piglit_gl_reinitialize_extensions->piglit_gl_invalidate_extensions
> >   util: move PIGLIT_PATH_SEP to piglit-util.h
> >   shader_runner: add ability to run multiple tests per process
> >
> >  framework/backends/json.py                             |  18 +-
> >  framework/options.py                                   |   1 +-
> >  framework/programs/run.py                              |  22 +-
> >  framework/results.py                                   |   8 +-
> >  framework/test/base.py                                 | 155 ++-
> >  framework/test/deqp.py                                 |   4 +-
> >  framework/test/opengl.py                               | 182 ++-
> >  framework/test/shader_test.py                          | 201 +++-
> >  piglit.conf.example                                    |   7 +-
> >  tests/all.py                                           |  34 +-
> >  tests/shader.py                                        |   4 +-
> >  tests/shaders/shader_runner.c                          | 560 +++++++---
> >  tests/util/piglit-framework-gl.c                       |   4 +-
> >  tests/util/piglit-framework-gl/piglit_glut_framework.c |   7 +-
> >  tests/util/piglit-framework-gl/piglit_wfl_framework.c  |   5 +-
> >  tests/util/piglit-util-gl.c                            |   2 +-
> >  tests/util/piglit-util-gl.h                            |   2 +-
> >  tests/util/piglit-util.c                               |   6 +-
> >  tests/util/piglit-util.h                               |   6 +-
> >  unittests/framework/backends/schema/piglit-9.json      | 122 ++-
> >  unittests/framework/backends/shared.py                 |   4 +-
> >  unittests/framework/backends/test_json_update.py       |  85 ++-
> >  unittests/framework/test/test_base.py                  | 138 +-
> >  unittests/framework/test/test_opengl.py                | 192 +--
> >  unittests/framework/test/test_shader_test.py           |  63 +-
> >  25 files changed, 1514 insertions(+), 318 deletions(-)
> >  create mode 100644 unittests/framework/backends/schema/piglit-9.json
> >
> > --
> > git-series 0.8.10


-- 
  Dylan Baker
  dylan at pnwbakers.com