[Piglit] [PATCH 00/35] Serialize profiles into XML at build time

Dylan Baker dylan at pnwbakers.com
Tue Apr 17 15:30:38 UTC 2018


Quoting Dylan Baker (2018-04-04 15:26:48)
> I don't expect everyone I've CC'd to give thorough review (or any
> review), I've mostly CC'd people who I think would be interested in this
> work, or who's work flow I might be altered by it.
> 
> Piglit has struggled to cope with the growing number of tests that it
> contains, especially with startup time. Piglit has always calculated
> tests at runtime, which was not a problem when there were only a few
> hundred or even thousand tests. Piglit now has roughly 55,000
> OpenGL/OpenGL ES tests, which is a lot to calculate at start up. It also
> means that piglit needs to keep a python object for each of those tests
> in memory, which has sent the resident memory usage soaring. We've also
> moved to automatic test discovery for glslparser, asmparser, and shader
> tests, which is very convenient and reduces typing, but further
> increases the amount of time spent starting up. This has even made
> features which decrease runtime, like fast skipping, hurt startup
> performance, making it a less than desirable tradeoff in some cases.
> Even on a relatively fast machine with an nvme disk 15-20 seconds is not
> an unheard of startup time. That might be okay to run 55,000 tests, but
> not if you only need a dozen, such as when bisecting.
> 
> This series is my proposal to fix that, mainly by moving much of that
> cost to build time. This series creates the infrastructure build XML
> base profiles at build time, which are installed with piglit instead of
> the python profiles. These profiles are lazily iterated over to ease
> memory usage, test objects are created as they are run, and python can
> garbage collect them as soon as they are done running. Along with that
> any filters applied to profiles (like removing 80% of the vs_in shader
> tests in quick) are done before the profile is serialized, and all fast
> skipping information is collected at build time as well, and encoded in
> the XML. All this means that start times are vastly reduced.
> 
> For example:
>     XML profiles
>     quick: 0.5
>     shader: 0.5
> 
>     master
>     quick: 11.6
>     shader: 7.3
> 
> This series also implements some optimizations for running without
> filters or test-lists, if you add a filter quick would take 2.5
> seconds, because that is necessary to calculate the total number of
> tests before starting.
> 
> To keep classic profiles like all, quick, quick_cl, gpu, cpu, and
> llvmpipe working this series adds meta profiles, small XML snippets that
> list other profiles. These can contain other meta profiles, xml
> profiles, or python profiles. This means that for most uses cases your
> existing command line will still work, `./piglit run quick out -c` will
> still do exactly the same thing as before, just faster.
> 
> The XML generated is dumb, there is no encoding of options or logic. An
> early version of this series did contain logic and options, but the
> result was pretty terrible. It was very hard to read, and the code to
> handle it was very complicated. I've chosen not to go down that path.
> There are drawbacks, some things that relied on run time generation have
> cannot be handled the same way, among them the "multi shader" concept,
> where shader_runner consumes a directory of shader_tests at a time. This
> was previously handled via a --process-isolation=false flag, now its
> encoded into profiles, "shader_multi" and "quick_shader_multi"; there
> was also an option to use glslparsertest with ES shaders and
> ARB_ES_compatibility, that is now "glslparser_arb_compat". I haven't
> added metaprofiles for these cases, although we certainly could (or you
> can write your own, the schema is dead simple), so `./piglit run quick
> out --process-isolation=false` is now `./piglit run quick_gl glslparser
> quick_shader_multi out`.
> 
> I've run this through our CI extensively, and gotten green results out
> of it across the board.
> 
> I know this is a big series, but piglit makes a lot of assumptions about the
> test profiles being created at runtime, and we've had to changes those
> assumptions.
> 
> 
> Dylan Baker (35):
>   update git ignore for this series
>   test/piglit_test: add ROOT_DIR variable
>   framework/profile: Allow a group manager class to be overwritten
>   framework/test: Use getter for altering PiglitBaseTest Command
>   framework/test: expose required and excluded platforms
>   framework/profile: Add a __len__ method to TestProfile
>   framework: Use custom class for ASM parser tests
>   framework/test: add a test class for built-in constants
>   tests: use BuiltInConstantsClass
>   framework: use a class method for building test via parsing
>   framework: do the same for shader test
>   framework/test: Split multishader too
>   framework/test/piglit_test: make cl_concurrency always a boolean
>   framework/test: Add class for cl-program-tester
>   framework/test: Make shader paths relative
>   framework/test: use relative paths for GLSLParser tests
>   tests/all: Make asmparser tests path relative
>   framework/test: make BuiltInConstantTest files relative
>   framework/test: make CLProgramTester take relative paths
>   profile: Add support for loading xml based profiles
>   profile: allow forcing python or xml loading
>   framework/profile: Add support for meta profiles
>   py_modules: Add support for out of tree builds
>   tests/quick: fix filtering of vs_in shader tests
>   tests: use meta profiles
>   shader_tests: correctly generate xml during out of tree builds
>   tests/glsl_parser_test.py: fix is_skip for serialized profiles
>   fix glslparser test for out of tree builds
>   fix asmparser test serialization for out of tree builds
>   tests/cl.py: fix out of tree serialization
>   opengl.py: Remove exported gl extensions
>   Add script to serialize profiles to XML
>   tests: Add script to find all hand written test files
>   Generate xml for builtin profiles
>   profile: use gz to compress profiles
> 
>  .gitignore                                        |    4 +-
>  CMakeLists.txt                                    |   10 +-
>  framework/profile.py                              |  203 +-
>  framework/test/glsl_parser_test.py                |   60 +-
>  framework/test/piglit_test.py                     |   68 +-
>  framework/test/shader_test.py                     |  131 +-
>  tests/CMakeLists.no_api.txt                       |   89 +-
>  tests/CMakeLists.txt                              |    2 +-
>  tests/all.meta.xml                                |    6 +-
>  tests/all.py                                      | 4999 +--------------
>  tests/cl.py                                       |   71 +-
>  tests/cpu.meta.xml                                |    4 +-
>  tests/cpu.py                                      |   31 +-
>  tests/find_static_tests.py                        |   74 +-
>  tests/glslparser.py                               |   63 +-
>  tests/gpu.meta.xml                                |    5 +-
>  tests/gpu.py                                      |   18 +-
>  tests/llvmpipe.meta.xml                           |    5 +-
>  tests/llvmpipe.py                                 |   33 +-
>  tests/llvmpipe_gl.py                              |   33 +-
>  tests/no_error.py                                 |   17 +-
>  tests/opencl_foreign.py                           |   34 +-
>  tests/opengl.py                                   | 4837 ++++++++++++++-
>  tests/py_modules/constants.py                     |    8 +-
>  tests/quick.meta.xml                              |    6 +-
>  tests/quick.py                                    |   97 +-
>  tests/quick_cl.meta.xml                           |    5 +-
>  tests/quick_cl.py                                 |   37 +-
>  tests/quick_gl.py                                 |   78 +-
>  tests/quick_shader.py                             |   53 +-
>  tests/serializer.py                               |  151 +-
>  tests/shader.py                                   |   71 +-
>  unittests/framework/test/test_glsl_parser_test.py |   61 +-
>  unittests/framework/test/test_shader_test.py      |   37 +-
>  unittests/framework/test_profile.py               |    5 +-
>  35 files changed, 6009 insertions(+), 5397 deletions(-)
>  create mode 100644 tests/CMakeLists.no_api.txt
>  create mode 100644 tests/all.meta.xml
>  delete mode 100644 tests/all.py
>  create mode 100644 tests/cpu.meta.xml
>  delete mode 100644 tests/cpu.py
>  create mode 100644 tests/find_static_tests.py
>  create mode 100644 tests/gpu.meta.xml
>  delete mode 100644 tests/gpu.py
>  create mode 100644 tests/llvmpipe.meta.xml
>  delete mode 100644 tests/llvmpipe.py
>  create mode 100644 tests/llvmpipe_gl.py
>  create mode 100644 tests/opencl_foreign.py
>  create mode 100644 tests/opengl.py
>  create mode 100644 tests/quick.meta.xml
>  delete mode 100644 tests/quick.py
>  create mode 100644 tests/quick_cl.meta.xml
>  delete mode 100644 tests/quick_cl.py
>  create mode 100644 tests/quick_gl.py
>  create mode 100644 tests/quick_shader.py
>  create mode 100644 tests/serializer.py
> 
> base-commit: 385fbbe2e73bf4f65f425101e8fa890ae5b52649
> -- 
> git-series 0.9.1
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/piglit

Does anyone have interest in reviewing or testing this?

Dylan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: signature
URL: <https://lists.freedesktop.org/archives/piglit/attachments/20180417/04da7d7e/attachment.sig>


More information about the Piglit mailing list