[Piglit] [PATCH 00/35] Serialize profiles into XML at build time

Dylan Baker dylan at pnwbakers.com
Wed Apr 4 22:26:48 UTC 2018

I don't expect everyone I've CC'd to give thorough review (or any
review), I've mostly CC'd people who I think would be interested in this
work, or who's work flow I might be altered by it.

Piglit has struggled to cope with the growing number of tests that it
contains, especially with startup time. Piglit has always calculated
tests at runtime, which was not a problem when there were only a few
hundred or even thousand tests. Piglit now has roughly 55,000
OpenGL/OpenGL ES tests, which is a lot to calculate at start up. It also
means that piglit needs to keep a python object for each of those tests
in memory, which has sent the resident memory usage soaring. We've also
moved to automatic test discovery for glslparser, asmparser, and shader
tests, which is very convenient and reduces typing, but further
increases the amount of time spent starting up. This has even made
features which decrease runtime, like fast skipping, hurt startup
performance, making it a less than desirable tradeoff in some cases.
Even on a relatively fast machine with an nvme disk 15-20 seconds is not
an unheard of startup time. That might be okay to run 55,000 tests, but
not if you only need a dozen, such as when bisecting.

This series is my proposal to fix that, mainly by moving much of that
cost to build time. This series creates the infrastructure build XML
base profiles at build time, which are installed with piglit instead of
the python profiles. These profiles are lazily iterated over to ease
memory usage, test objects are created as they are run, and python can
garbage collect them as soon as they are done running. Along with that
any filters applied to profiles (like removing 80% of the vs_in shader
tests in quick) are done before the profile is serialized, and all fast
skipping information is collected at build time as well, and encoded in
the XML. All this means that start times are vastly reduced.

For example:
    XML profiles
    quick: 0.5
    shader: 0.5

    quick: 11.6
    shader: 7.3

This series also implements some optimizations for running without
filters or test-lists, if you add a filter quick would take 2.5
seconds, because that is necessary to calculate the total number of
tests before starting.

To keep classic profiles like all, quick, quick_cl, gpu, cpu, and
llvmpipe working this series adds meta profiles, small XML snippets that
list other profiles. These can contain other meta profiles, xml
profiles, or python profiles. This means that for most uses cases your
existing command line will still work, `./piglit run quick out -c` will
still do exactly the same thing as before, just faster.

The XML generated is dumb, there is no encoding of options or logic. An
early version of this series did contain logic and options, but the
result was pretty terrible. It was very hard to read, and the code to
handle it was very complicated. I've chosen not to go down that path.
There are drawbacks, some things that relied on run time generation have
cannot be handled the same way, among them the "multi shader" concept,
where shader_runner consumes a directory of shader_tests at a time. This
was previously handled via a --process-isolation=false flag, now its
encoded into profiles, "shader_multi" and "quick_shader_multi"; there
was also an option to use glslparsertest with ES shaders and
ARB_ES_compatibility, that is now "glslparser_arb_compat". I haven't
added metaprofiles for these cases, although we certainly could (or you
can write your own, the schema is dead simple), so `./piglit run quick
out --process-isolation=false` is now `./piglit run quick_gl glslparser
quick_shader_multi out`.

I've run this through our CI extensively, and gotten green results out
of it across the board.

I know this is a big series, but piglit makes a lot of assumptions about the
test profiles being created at runtime, and we've had to changes those

Dylan Baker (35):
  update git ignore for this series
  test/piglit_test: add ROOT_DIR variable
  framework/profile: Allow a group manager class to be overwritten
  framework/test: Use getter for altering PiglitBaseTest Command
  framework/test: expose required and excluded platforms
  framework/profile: Add a __len__ method to TestProfile
  framework: Use custom class for ASM parser tests
  framework/test: add a test class for built-in constants
  tests: use BuiltInConstantsClass
  framework: use a class method for building test via parsing
  framework: do the same for shader test
  framework/test: Split multishader too
  framework/test/piglit_test: make cl_concurrency always a boolean
  framework/test: Add class for cl-program-tester
  framework/test: Make shader paths relative
  framework/test: use relative paths for GLSLParser tests
  tests/all: Make asmparser tests path relative
  framework/test: make BuiltInConstantTest files relative
  framework/test: make CLProgramTester take relative paths
  profile: Add support for loading xml based profiles
  profile: allow forcing python or xml loading
  framework/profile: Add support for meta profiles
  py_modules: Add support for out of tree builds
  tests/quick: fix filtering of vs_in shader tests
  tests: use meta profiles
  shader_tests: correctly generate xml during out of tree builds
  tests/glsl_parser_test.py: fix is_skip for serialized profiles
  fix glslparser test for out of tree builds
  fix asmparser test serialization for out of tree builds
  tests/cl.py: fix out of tree serialization
  opengl.py: Remove exported gl extensions
  Add script to serialize profiles to XML
  tests: Add script to find all hand written test files
  Generate xml for builtin profiles
  profile: use gz to compress profiles

 .gitignore                                        |    4 +-
 CMakeLists.txt                                    |   10 +-
 framework/profile.py                              |  203 +-
 framework/test/glsl_parser_test.py                |   60 +-
 framework/test/piglit_test.py                     |   68 +-
 framework/test/shader_test.py                     |  131 +-
 tests/CMakeLists.no_api.txt                       |   89 +-
 tests/CMakeLists.txt                              |    2 +-
 tests/all.meta.xml                                |    6 +-
 tests/all.py                                      | 4999 +--------------
 tests/cl.py                                       |   71 +-
 tests/cpu.meta.xml                                |    4 +-
 tests/cpu.py                                      |   31 +-
 tests/find_static_tests.py                        |   74 +-
 tests/glslparser.py                               |   63 +-
 tests/gpu.meta.xml                                |    5 +-
 tests/gpu.py                                      |   18 +-
 tests/llvmpipe.meta.xml                           |    5 +-
 tests/llvmpipe.py                                 |   33 +-
 tests/llvmpipe_gl.py                              |   33 +-
 tests/no_error.py                                 |   17 +-
 tests/opencl_foreign.py                           |   34 +-
 tests/opengl.py                                   | 4837 ++++++++++++++-
 tests/py_modules/constants.py                     |    8 +-
 tests/quick.meta.xml                              |    6 +-
 tests/quick.py                                    |   97 +-
 tests/quick_cl.meta.xml                           |    5 +-
 tests/quick_cl.py                                 |   37 +-
 tests/quick_gl.py                                 |   78 +-
 tests/quick_shader.py                             |   53 +-
 tests/serializer.py                               |  151 +-
 tests/shader.py                                   |   71 +-
 unittests/framework/test/test_glsl_parser_test.py |   61 +-
 unittests/framework/test/test_shader_test.py      |   37 +-
 unittests/framework/test_profile.py               |    5 +-
 35 files changed, 6009 insertions(+), 5397 deletions(-)
 create mode 100644 tests/CMakeLists.no_api.txt
 create mode 100644 tests/all.meta.xml
 delete mode 100644 tests/all.py
 create mode 100644 tests/cpu.meta.xml
 delete mode 100644 tests/cpu.py
 create mode 100644 tests/find_static_tests.py
 create mode 100644 tests/gpu.meta.xml
 delete mode 100644 tests/gpu.py
 create mode 100644 tests/llvmpipe.meta.xml
 delete mode 100644 tests/llvmpipe.py
 create mode 100644 tests/llvmpipe_gl.py
 create mode 100644 tests/opencl_foreign.py
 create mode 100644 tests/opengl.py
 create mode 100644 tests/quick.meta.xml
 delete mode 100644 tests/quick.py
 create mode 100644 tests/quick_cl.meta.xml
 delete mode 100644 tests/quick_cl.py
 create mode 100644 tests/quick_gl.py
 create mode 100644 tests/quick_shader.py
 create mode 100644 tests/serializer.py

base-commit: 385fbbe2e73bf4f65f425101e8fa890ae5b52649
git-series 0.9.1

More information about the Piglit mailing list