<div dir="ltr"><div><div>Is this use case affected? piglit run --deqp-mustpass-list --process-isolation 0 -p gbm -c quick cts_gl45 deqp_gles2 deqp_gles3 deqp_gles31 </div><div>Yes, that is just 1 command to run all those test suites at the same time. </div><div> </div>I use my personal "deqp" piglit branch that also disables process isolation for glcts and deqp, and parses the deqp mustpass lists which are in txt files. </div>Marek </div><div class="gmail_extra"> <div class="gmail_quote">On Wed, Apr 4, 2018 at 6:26 PM, Dylan Baker <<a href="mailto:dylan@pnwbakers.com" target="_blank">dylan@pnwbakers.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I don't expect everyone I've CC'd to give thorough review (or any review), I've mostly CC'd people who I think would be interested in this work, or who's work flow I might be altered by it. Piglit has struggled to cope with the growing number of tests that it contains, especially with startup time. Piglit has always calculated tests at runtime, which was not a problem when there were only a few hundred or even thousand tests. Piglit now has roughly 55,000 OpenGL/OpenGL ES tests, which is a lot to calculate at start up. It also means that piglit needs to keep a python object for each of those tests in memory, which has sent the resident memory usage soaring. We've also moved to automatic test discovery for glslparser, asmparser, and shader tests, which is very convenient and reduces typing, but further increases the amount of time spent starting up. This has even made features which decrease runtime, like fast skipping, hurt startup performance, making it a less than desirable tradeoff in some cases. Even on a relatively fast machine with an nvme disk 15-20 seconds is not an unheard of startup time. That might be okay to run 55,000 tests, but not if you only need a dozen, such as when bisecting. This series is my proposal to fix that, mainly by moving much of that cost to build time. This series creates the infrastructure build XML base profiles at build time, which are installed with piglit instead of the python profiles. These profiles are lazily iterated over to ease memory usage, test objects are created as they are run, and python can garbage collect them as soon as they are done running. Along with that any filters applied to profiles (like removing 80% of the vs_in shader tests in quick) are done before the profile is serialized, and all fast skipping information is collected at build time as well, and encoded in the XML. All this means that start times are vastly reduced. For example: XML profiles quick: 0.5 shader: 0.5 master quick: 11.6 shader: 7.3 This series also implements some optimizations for running without filters or test-lists, if you add a filter quick would take 2.5 seconds, because that is necessary to calculate the total number of tests before starting. To keep classic profiles like all, quick, quick_cl, gpu, cpu, and llvmpipe working this series adds meta profiles, small XML snippets that list other profiles. These can contain other meta profiles, xml profiles, or python profiles. This means that for most uses cases your existing command line will still work, `./piglit run quick out -c` will still do exactly the same thing as before, just faster. The XML generated is dumb, there is no encoding of options or logic. An early version of this series did contain logic and options, but the result was pretty terrible. It was very hard to read, and the code to handle it was very complicated. I've chosen not to go down that path. There are drawbacks, some things that relied on run time generation have cannot be handled the same way, among them the "multi shader" concept, where shader_runner consumes a directory of shader_tests at a time. This was previously handled via a --process-isolation=false flag, now its encoded into profiles, "shader_multi" and "quick_shader_multi"; there was also an option to use glslparsertest with ES shaders and ARB_ES_compatibility, that is now "glslparser_arb_compat". I haven't added metaprofiles for these cases, although we certainly could (or you can write your own, the schema is dead simple), so `./piglit run quick out --process-isolation=false` is now `./piglit run quick_gl glslparser quick_shader_multi out`. I've run this through our CI extensively, and gotten green results out of it across the board. I know this is a big series, but piglit makes a lot of assumptions about the test profiles being created at runtime, and we've had to changes those assumptions. Dylan Baker (35): update git ignore for this series test/piglit_test: add ROOT_DIR variable framework/profile: Allow a group manager class to be overwritten framework/test: Use getter for altering PiglitBaseTest Command framework/test: expose required and excluded platforms framework/profile: Add a __len__ method to TestProfile framework: Use custom class for ASM parser tests framework/test: add a test class for built-in constants tests: use BuiltInConstantsClass framework: use a class method for building test via parsing framework: do the same for shader test framework/test: Split multishader too framework/test/piglit_test: make cl_concurrency always a boolean framework/test: Add class for cl-program-tester framework/test: Make shader paths relative framework/test: use relative paths for GLSLParser tests tests/all: Make asmparser tests path relative framework/test: make BuiltInConstantTest files relative framework/test: make CLProgramTester take relative paths profile: Add support for loading xml based profiles profile: allow forcing python or xml loading framework/profile: Add support for meta profiles py_modules: Add support for out of tree builds tests/quick: fix filtering of vs_in shader tests tests: use meta profiles shader_tests: correctly generate xml during out of tree builds tests/glsl_parser_test.py: fix is_skip for serialized profiles fix glslparser test for out of tree builds fix asmparser test serialization for out of tree builds tests/cl.py: fix out of tree serialization opengl.py: Remove exported gl extensions Add script to serialize profiles to XML tests: Add script to find all hand written test files Generate xml for builtin profiles profile: use gz to compress profiles .gitignore | 4 +- CMakeLists.txt | 10 +- framework/profile.py | 203 +- framework/test/glsl_parser_test.py | 60 +- framework/test/piglit_test.py | 68 +- framework/test/shader_test.py | 131 +- tests/CMakeLists.no_api.txt | 89 +- tests/CMakeLists.txt | 2 +- tests/all.meta.xml | 6 +- tests/all.py | 4999 +-------------- tests/cl.py | 71 +- tests/cpu.meta.xml | 4 +- tests/cpu.py | 31 +- tests/find_static_tests.py | 74 +- tests/glslparser.py | 63 +- tests/gpu.meta.xml | 5 +- tests/gpu.py | 18 +- tests/llvmpipe.meta.xml | 5 +- tests/llvmpipe.py | 33 +- tests/llvmpipe_gl.py | 33 +- tests/no_error.py | 17 +- tests/opencl_foreign.py | 34 +- tests/opengl.py | 4837 ++++++++++++++- tests/py_modules/constants.py | 8 +- tests/quick.meta.xml | 6 +- tests/quick.py | 97 +- tests/quick_cl.meta.xml | 5 +- tests/quick_cl.py | 37 +- tests/quick_gl.py | 78 +- tests/quick_shader.py | 53 +- tests/serializer.py | 151 +- tests/shader.py | 71 +- unittests/framework/test/test_glsl_parser_test.py | 61 +- unittests/framework/test/test_shader_test.py | 37 +- unittests/framework/test_profile.py | 5 +- 35 files changed, 6009 insertions(+), 5397 deletions(-) create mode 100644 tests/CMakeLists.no_api.txt create mode 100644 tests/all.meta.xml delete mode 100644 tests/all.py create mode 100644 tests/cpu.meta.xml delete mode 100644 tests/cpu.py create mode 100644 tests/find_static_tests.py create mode 100644 tests/gpu.meta.xml delete mode 100644 tests/gpu.py create mode 100644 tests/llvmpipe.meta.xml delete mode 100644 tests/llvmpipe.py create mode 100644 tests/llvmpipe_gl.py create mode 100644 tests/opencl_foreign.py create mode 100644 tests/opengl.py create mode 100644 tests/quick.meta.xml delete mode 100644 tests/quick.py create mode 100644 tests/quick_cl.meta.xml delete mode 100644 tests/quick_cl.py create mode 100644 tests/quick_gl.py create mode 100644 tests/quick_shader.py create mode 100644 tests/serializer.py base-commit: 385fbbe2e73bf4f65f425101e8fa890ae5b52649 -- git-series 0.9.1 </blockquote></div> </div>