[Piglit] [PATCH 00/35] Serialize profiles into XML at build time
Marek Olšák
maraeo at gmail.com
Wed Apr 4 23:37:47 UTC 2018
Is this use case affected?
piglit run --deqp-mustpass-list --process-isolation 0 -p gbm -c quick
cts_gl45 deqp_gles2 deqp_gles3 deqp_gles31
Yes, that is just 1 command to run all those test suites at the same time.
I use my personal "deqp" piglit branch that also disables process isolation
for glcts and deqp, and parses the deqp mustpass lists which are in txt
files.
Marek
On Wed, Apr 4, 2018 at 6:26 PM, Dylan Baker <dylan at pnwbakers.com> wrote:
> I don't expect everyone I've CC'd to give thorough review (or any
> review), I've mostly CC'd people who I think would be interested in this
> work, or who's work flow I might be altered by it.
>
> Piglit has struggled to cope with the growing number of tests that it
> contains, especially with startup time. Piglit has always calculated
> tests at runtime, which was not a problem when there were only a few
> hundred or even thousand tests. Piglit now has roughly 55,000
> OpenGL/OpenGL ES tests, which is a lot to calculate at start up. It also
> means that piglit needs to keep a python object for each of those tests
> in memory, which has sent the resident memory usage soaring. We've also
> moved to automatic test discovery for glslparser, asmparser, and shader
> tests, which is very convenient and reduces typing, but further
> increases the amount of time spent starting up. This has even made
> features which decrease runtime, like fast skipping, hurt startup
> performance, making it a less than desirable tradeoff in some cases.
> Even on a relatively fast machine with an nvme disk 15-20 seconds is not
> an unheard of startup time. That might be okay to run 55,000 tests, but
> not if you only need a dozen, such as when bisecting.
>
> This series is my proposal to fix that, mainly by moving much of that
> cost to build time. This series creates the infrastructure build XML
> base profiles at build time, which are installed with piglit instead of
> the python profiles. These profiles are lazily iterated over to ease
> memory usage, test objects are created as they are run, and python can
> garbage collect them as soon as they are done running. Along with that
> any filters applied to profiles (like removing 80% of the vs_in shader
> tests in quick) are done before the profile is serialized, and all fast
> skipping information is collected at build time as well, and encoded in
> the XML. All this means that start times are vastly reduced.
>
> For example:
> XML profiles
> quick: 0.5
> shader: 0.5
>
> master
> quick: 11.6
> shader: 7.3
>
> This series also implements some optimizations for running without
> filters or test-lists, if you add a filter quick would take 2.5
> seconds, because that is necessary to calculate the total number of
> tests before starting.
>
> To keep classic profiles like all, quick, quick_cl, gpu, cpu, and
> llvmpipe working this series adds meta profiles, small XML snippets that
> list other profiles. These can contain other meta profiles, xml
> profiles, or python profiles. This means that for most uses cases your
> existing command line will still work, `./piglit run quick out -c` will
> still do exactly the same thing as before, just faster.
>
> The XML generated is dumb, there is no encoding of options or logic. An
> early version of this series did contain logic and options, but the
> result was pretty terrible. It was very hard to read, and the code to
> handle it was very complicated. I've chosen not to go down that path.
> There are drawbacks, some things that relied on run time generation have
> cannot be handled the same way, among them the "multi shader" concept,
> where shader_runner consumes a directory of shader_tests at a time. This
> was previously handled via a --process-isolation=false flag, now its
> encoded into profiles, "shader_multi" and "quick_shader_multi"; there
> was also an option to use glslparsertest with ES shaders and
> ARB_ES_compatibility, that is now "glslparser_arb_compat". I haven't
> added metaprofiles for these cases, although we certainly could (or you
> can write your own, the schema is dead simple), so `./piglit run quick
> out --process-isolation=false` is now `./piglit run quick_gl glslparser
> quick_shader_multi out`.
>
> I've run this through our CI extensively, and gotten green results out
> of it across the board.
>
> I know this is a big series, but piglit makes a lot of assumptions about
> the
> test profiles being created at runtime, and we've had to changes those
> assumptions.
>
>
> Dylan Baker (35):
> update git ignore for this series
> test/piglit_test: add ROOT_DIR variable
> framework/profile: Allow a group manager class to be overwritten
> framework/test: Use getter for altering PiglitBaseTest Command
> framework/test: expose required and excluded platforms
> framework/profile: Add a __len__ method to TestProfile
> framework: Use custom class for ASM parser tests
> framework/test: add a test class for built-in constants
> tests: use BuiltInConstantsClass
> framework: use a class method for building test via parsing
> framework: do the same for shader test
> framework/test: Split multishader too
> framework/test/piglit_test: make cl_concurrency always a boolean
> framework/test: Add class for cl-program-tester
> framework/test: Make shader paths relative
> framework/test: use relative paths for GLSLParser tests
> tests/all: Make asmparser tests path relative
> framework/test: make BuiltInConstantTest files relative
> framework/test: make CLProgramTester take relative paths
> profile: Add support for loading xml based profiles
> profile: allow forcing python or xml loading
> framework/profile: Add support for meta profiles
> py_modules: Add support for out of tree builds
> tests/quick: fix filtering of vs_in shader tests
> tests: use meta profiles
> shader_tests: correctly generate xml during out of tree builds
> tests/glsl_parser_test.py: fix is_skip for serialized profiles
> fix glslparser test for out of tree builds
> fix asmparser test serialization for out of tree builds
> tests/cl.py: fix out of tree serialization
> opengl.py: Remove exported gl extensions
> Add script to serialize profiles to XML
> tests: Add script to find all hand written test files
> Generate xml for builtin profiles
> profile: use gz to compress profiles
>
> .gitignore | 4 +-
> CMakeLists.txt | 10 +-
> framework/profile.py | 203 +-
> framework/test/glsl_parser_test.py | 60 +-
> framework/test/piglit_test.py | 68 +-
> framework/test/shader_test.py | 131 +-
> tests/CMakeLists.no_api.txt | 89 +-
> tests/CMakeLists.txt | 2 +-
> tests/all.meta.xml | 6 +-
> tests/all.py | 4999 +--------------
> tests/cl.py | 71 +-
> tests/cpu.meta.xml | 4 +-
> tests/cpu.py | 31 +-
> tests/find_static_tests.py | 74 +-
> tests/glslparser.py | 63 +-
> tests/gpu.meta.xml | 5 +-
> tests/gpu.py | 18 +-
> tests/llvmpipe.meta.xml | 5 +-
> tests/llvmpipe.py | 33 +-
> tests/llvmpipe_gl.py | 33 +-
> tests/no_error.py | 17 +-
> tests/opencl_foreign.py | 34 +-
> tests/opengl.py | 4837 ++++++++++++++-
> tests/py_modules/constants.py | 8 +-
> tests/quick.meta.xml | 6 +-
> tests/quick.py | 97 +-
> tests/quick_cl.meta.xml | 5 +-
> tests/quick_cl.py | 37 +-
> tests/quick_gl.py | 78 +-
> tests/quick_shader.py | 53 +-
> tests/serializer.py | 151 +-
> tests/shader.py | 71 +-
> unittests/framework/test/test_glsl_parser_test.py | 61 +-
> unittests/framework/test/test_shader_test.py | 37 +-
> unittests/framework/test_profile.py | 5 +-
> 35 files changed, 6009 insertions(+), 5397 deletions(-)
> create mode 100644 tests/CMakeLists.no_api.txt
> create mode 100644 tests/all.meta.xml
> delete mode 100644 tests/all.py
> create mode 100644 tests/cpu.meta.xml
> delete mode 100644 tests/cpu.py
> create mode 100644 tests/find_static_tests.py
> create mode 100644 tests/gpu.meta.xml
> delete mode 100644 tests/gpu.py
> create mode 100644 tests/llvmpipe.meta.xml
> delete mode 100644 tests/llvmpipe.py
> create mode 100644 tests/llvmpipe_gl.py
> create mode 100644 tests/opencl_foreign.py
> create mode 100644 tests/opengl.py
> create mode 100644 tests/quick.meta.xml
> delete mode 100644 tests/quick.py
> create mode 100644 tests/quick_cl.meta.xml
> delete mode 100644 tests/quick_cl.py
> create mode 100644 tests/quick_gl.py
> create mode 100644 tests/quick_shader.py
> create mode 100644 tests/serializer.py
>
> base-commit: 385fbbe2e73bf4f65f425101e8fa890ae5b52649
> --
> git-series 0.9.1
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/piglit/attachments/20180404/02c08037/attachment.html>
More information about the Piglit
mailing list