[Piglit] [PATCH 00/35] Serialize profiles into XML at build time
Dylan Baker
dylan at pnwbakers.com
Wed Apr 4 22:32:55 UTC 2018
Quoting Dylan Baker (2018-04-04 15:26:48)
> I don't expect everyone I've CC'd to give thorough review (or any
> review), I've mostly CC'd people who I think would be interested in this
> work, or who's work flow I might be altered by it.
>
> Piglit has struggled to cope with the growing number of tests that it
> contains, especially with startup time. Piglit has always calculated
> tests at runtime, which was not a problem when there were only a few
> hundred or even thousand tests. Piglit now has roughly 55,000
> OpenGL/OpenGL ES tests, which is a lot to calculate at start up. It also
> means that piglit needs to keep a python object for each of those tests
> in memory, which has sent the resident memory usage soaring. We've also
> moved to automatic test discovery for glslparser, asmparser, and shader
> tests, which is very convenient and reduces typing, but further
> increases the amount of time spent starting up. This has even made
> features which decrease runtime, like fast skipping, hurt startup
> performance, making it a less than desirable tradeoff in some cases.
> Even on a relatively fast machine with an nvme disk 15-20 seconds is not
> an unheard of startup time. That might be okay to run 55,000 tests, but
> not if you only need a dozen, such as when bisecting.
>
> This series is my proposal to fix that, mainly by moving much of that
> cost to build time. This series creates the infrastructure build XML
> base profiles at build time, which are installed with piglit instead of
> the python profiles. These profiles are lazily iterated over to ease
> memory usage, test objects are created as they are run, and python can
> garbage collect them as soon as they are done running. Along with that
> any filters applied to profiles (like removing 80% of the vs_in shader
> tests in quick) are done before the profile is serialized, and all fast
> skipping information is collected at build time as well, and encoded in
> the XML. All this means that start times are vastly reduced.
>
> For example:
> XML profiles
> quick: 0.5
> shader: 0.5
>
> master
> quick: 11.6
> shader: 7.3
>
> This series also implements some optimizations for running without
> filters or test-lists, if you add a filter quick would take 2.5
> seconds, because that is necessary to calculate the total number of
> tests before starting.
>
> To keep classic profiles like all, quick, quick_cl, gpu, cpu, and
> llvmpipe working this series adds meta profiles, small XML snippets that
> list other profiles. These can contain other meta profiles, xml
> profiles, or python profiles. This means that for most uses cases your
> existing command line will still work, `./piglit run quick out -c` will
> still do exactly the same thing as before, just faster.
>
> The XML generated is dumb, there is no encoding of options or logic. An
> early version of this series did contain logic and options, but the
> result was pretty terrible. It was very hard to read, and the code to
> handle it was very complicated. I've chosen not to go down that path.
> There are drawbacks, some things that relied on run time generation have
> cannot be handled the same way, among them the "multi shader" concept,
> where shader_runner consumes a directory of shader_tests at a time. This
> was previously handled via a --process-isolation=false flag, now its
> encoded into profiles, "shader_multi" and "quick_shader_multi"; there
> was also an option to use glslparsertest with ES shaders and
> ARB_ES_compatibility, that is now "glslparser_arb_compat". I haven't
> added metaprofiles for these cases, although we certainly could (or you
> can write your own, the schema is dead simple), so `./piglit run quick
> out --process-isolation=false` is now `./piglit run quick_gl glslparser
> quick_shader_multi out`.
>
> I've run this through our CI extensively, and gotten green results out
> of it across the board.
>
> I know this is a big series, but piglit makes a lot of assumptions about the
> test profiles being created at runtime, and we've had to changes those
> assumptions.
>
>
> Dylan Baker (35):
> update git ignore for this series
> test/piglit_test: add ROOT_DIR variable
> framework/profile: Allow a group manager class to be overwritten
> framework/test: Use getter for altering PiglitBaseTest Command
> framework/test: expose required and excluded platforms
> framework/profile: Add a __len__ method to TestProfile
> framework: Use custom class for ASM parser tests
> framework/test: add a test class for built-in constants
> tests: use BuiltInConstantsClass
> framework: use a class method for building test via parsing
> framework: do the same for shader test
> framework/test: Split multishader too
> framework/test/piglit_test: make cl_concurrency always a boolean
> framework/test: Add class for cl-program-tester
> framework/test: Make shader paths relative
> framework/test: use relative paths for GLSLParser tests
> tests/all: Make asmparser tests path relative
> framework/test: make BuiltInConstantTest files relative
> framework/test: make CLProgramTester take relative paths
> profile: Add support for loading xml based profiles
> profile: allow forcing python or xml loading
> framework/profile: Add support for meta profiles
> py_modules: Add support for out of tree builds
> tests/quick: fix filtering of vs_in shader tests
> tests: use meta profiles
> shader_tests: correctly generate xml during out of tree builds
> tests/glsl_parser_test.py: fix is_skip for serialized profiles
> fix glslparser test for out of tree builds
> fix asmparser test serialization for out of tree builds
> tests/cl.py: fix out of tree serialization
> opengl.py: Remove exported gl extensions
> Add script to serialize profiles to XML
> tests: Add script to find all hand written test files
> Generate xml for builtin profiles
> profile: use gz to compress profiles
>
> .gitignore | 4 +-
> CMakeLists.txt | 10 +-
> framework/profile.py | 203 +-
> framework/test/glsl_parser_test.py | 60 +-
> framework/test/piglit_test.py | 68 +-
> framework/test/shader_test.py | 131 +-
> tests/CMakeLists.no_api.txt | 89 +-
> tests/CMakeLists.txt | 2 +-
> tests/all.meta.xml | 6 +-
> tests/all.py | 4999 +--------------
> tests/cl.py | 71 +-
> tests/cpu.meta.xml | 4 +-
> tests/cpu.py | 31 +-
> tests/find_static_tests.py | 74 +-
> tests/glslparser.py | 63 +-
> tests/gpu.meta.xml | 5 +-
> tests/gpu.py | 18 +-
> tests/llvmpipe.meta.xml | 5 +-
> tests/llvmpipe.py | 33 +-
> tests/llvmpipe_gl.py | 33 +-
> tests/no_error.py | 17 +-
> tests/opencl_foreign.py | 34 +-
> tests/opengl.py | 4837 ++++++++++++++-
> tests/py_modules/constants.py | 8 +-
> tests/quick.meta.xml | 6 +-
> tests/quick.py | 97 +-
> tests/quick_cl.meta.xml | 5 +-
> tests/quick_cl.py | 37 +-
> tests/quick_gl.py | 78 +-
> tests/quick_shader.py | 53 +-
> tests/serializer.py | 151 +-
> tests/shader.py | 71 +-
> unittests/framework/test/test_glsl_parser_test.py | 61 +-
> unittests/framework/test/test_shader_test.py | 37 +-
> unittests/framework/test_profile.py | 5 +-
> 35 files changed, 6009 insertions(+), 5397 deletions(-)
> create mode 100644 tests/CMakeLists.no_api.txt
> create mode 100644 tests/all.meta.xml
> delete mode 100644 tests/all.py
> create mode 100644 tests/cpu.meta.xml
> delete mode 100644 tests/cpu.py
> create mode 100644 tests/find_static_tests.py
> create mode 100644 tests/gpu.meta.xml
> delete mode 100644 tests/gpu.py
> create mode 100644 tests/llvmpipe.meta.xml
> delete mode 100644 tests/llvmpipe.py
> create mode 100644 tests/llvmpipe_gl.py
> create mode 100644 tests/opencl_foreign.py
> create mode 100644 tests/opengl.py
> create mode 100644 tests/quick.meta.xml
> delete mode 100644 tests/quick.py
> create mode 100644 tests/quick_cl.meta.xml
> delete mode 100644 tests/quick_cl.py
> create mode 100644 tests/quick_gl.py
> create mode 100644 tests/quick_shader.py
> create mode 100644 tests/serializer.py
>
> base-commit: 385fbbe2e73bf4f65f425101e8fa890ae5b52649
> --
> git-series 0.9.1
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/piglit
Since I'm sure that some of these will get held up in moderation, or if you
don't want to git-apply 35 patches :), a branch is here:
github.com/dcbaker/piglit xml-profile
Dylan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: signature
URL: <https://lists.freedesktop.org/archives/piglit/attachments/20180404/de142f9c/attachment.sig>
More information about the Piglit
mailing list