[Piglit] [PATCH 00/35] Serialize profiles into XML at build time
Juan A. Suarez Romero
jasuarez at igalia.com
Wed May 2 16:49:08 UTC 2018
Hi, Dylan.
I see you've pushed this series.
Now, when I'm trying to run some profiles (mainly, tests/crucible and
tests/khr_gl* ), seems they are broken:
[0000/7776]
Traceback (most recent call last):
File "./piglit", line 178, in <module>
main()
File "./piglit", line 174, in main
sys.exit(runner(args))
File "/home/igalia/jasuarez/piglit/framework/exceptions.py", line 51, in
_inner
func(*args, **kwargs)
File "/home/igalia/jasuarez/piglit/framework/programs/run.py", line 370, in
run
backend.finalize({'time_elapsed': time_elapsed.to_json()})
File "/home/igalia/jasuarez/piglit/framework/backends/json.py", line 163, in
finalize
assert data['tests']
AssertionError
J.A.
On Tue, 2018-04-17 at 08:30 -0700, Dylan Baker wrote:
> Quoting Dylan Baker (2018-04-04 15:26:48)
> > I don't expect everyone I've CC'd to give thorough review (or any
> > review), I've mostly CC'd people who I think would be interested in this
> > work, or who's work flow I might be altered by it.
> >
> > Piglit has struggled to cope with the growing number of tests that it
> > contains, especially with startup time. Piglit has always calculated
> > tests at runtime, which was not a problem when there were only a few
> > hundred or even thousand tests. Piglit now has roughly 55,000
> > OpenGL/OpenGL ES tests, which is a lot to calculate at start up. It also
> > means that piglit needs to keep a python object for each of those tests
> > in memory, which has sent the resident memory usage soaring. We've also
> > moved to automatic test discovery for glslparser, asmparser, and shader
> > tests, which is very convenient and reduces typing, but further
> > increases the amount of time spent starting up. This has even made
> > features which decrease runtime, like fast skipping, hurt startup
> > performance, making it a less than desirable tradeoff in some cases.
> > Even on a relatively fast machine with an nvme disk 15-20 seconds is not
> > an unheard of startup time. That might be okay to run 55,000 tests, but
> > not if you only need a dozen, such as when bisecting.
> >
> > This series is my proposal to fix that, mainly by moving much of that
> > cost to build time. This series creates the infrastructure build XML
> > base profiles at build time, which are installed with piglit instead of
> > the python profiles. These profiles are lazily iterated over to ease
> > memory usage, test objects are created as they are run, and python can
> > garbage collect them as soon as they are done running. Along with that
> > any filters applied to profiles (like removing 80% of the vs_in shader
> > tests in quick) are done before the profile is serialized, and all fast
> > skipping information is collected at build time as well, and encoded in
> > the XML. All this means that start times are vastly reduced.
> >
> > For example:
> > XML profiles
> > quick: 0.5
> > shader: 0.5
> >
> > master
> > quick: 11.6
> > shader: 7.3
> >
> > This series also implements some optimizations for running without
> > filters or test-lists, if you add a filter quick would take 2.5
> > seconds, because that is necessary to calculate the total number of
> > tests before starting.
> >
> > To keep classic profiles like all, quick, quick_cl, gpu, cpu, and
> > llvmpipe working this series adds meta profiles, small XML snippets that
> > list other profiles. These can contain other meta profiles, xml
> > profiles, or python profiles. This means that for most uses cases your
> > existing command line will still work, `./piglit run quick out -c` will
> > still do exactly the same thing as before, just faster.
> >
> > The XML generated is dumb, there is no encoding of options or logic. An
> > early version of this series did contain logic and options, but the
> > result was pretty terrible. It was very hard to read, and the code to
> > handle it was very complicated. I've chosen not to go down that path.
> > There are drawbacks, some things that relied on run time generation have
> > cannot be handled the same way, among them the "multi shader" concept,
> > where shader_runner consumes a directory of shader_tests at a time. This
> > was previously handled via a --process-isolation=false flag, now its
> > encoded into profiles, "shader_multi" and "quick_shader_multi"; there
> > was also an option to use glslparsertest with ES shaders and
> > ARB_ES_compatibility, that is now "glslparser_arb_compat". I haven't
> > added metaprofiles for these cases, although we certainly could (or you
> > can write your own, the schema is dead simple), so `./piglit run quick
> > out --process-isolation=false` is now `./piglit run quick_gl glslparser
> > quick_shader_multi out`.
> >
> > I've run this through our CI extensively, and gotten green results out
> > of it across the board.
> >
> > I know this is a big series, but piglit makes a lot of assumptions about the
> > test profiles being created at runtime, and we've had to changes those
> > assumptions.
> >
> >
> > Dylan Baker (35):
> > update git ignore for this series
> > test/piglit_test: add ROOT_DIR variable
> > framework/profile: Allow a group manager class to be overwritten
> > framework/test: Use getter for altering PiglitBaseTest Command
> > framework/test: expose required and excluded platforms
> > framework/profile: Add a __len__ method to TestProfile
> > framework: Use custom class for ASM parser tests
> > framework/test: add a test class for built-in constants
> > tests: use BuiltInConstantsClass
> > framework: use a class method for building test via parsing
> > framework: do the same for shader test
> > framework/test: Split multishader too
> > framework/test/piglit_test: make cl_concurrency always a boolean
> > framework/test: Add class for cl-program-tester
> > framework/test: Make shader paths relative
> > framework/test: use relative paths for GLSLParser tests
> > tests/all: Make asmparser tests path relative
> > framework/test: make BuiltInConstantTest files relative
> > framework/test: make CLProgramTester take relative paths
> > profile: Add support for loading xml based profiles
> > profile: allow forcing python or xml loading
> > framework/profile: Add support for meta profiles
> > py_modules: Add support for out of tree builds
> > tests/quick: fix filtering of vs_in shader tests
> > tests: use meta profiles
> > shader_tests: correctly generate xml during out of tree builds
> > tests/glsl_parser_test.py: fix is_skip for serialized profiles
> > fix glslparser test for out of tree builds
> > fix asmparser test serialization for out of tree builds
> > tests/cl.py: fix out of tree serialization
> > opengl.py: Remove exported gl extensions
> > Add script to serialize profiles to XML
> > tests: Add script to find all hand written test files
> > Generate xml for builtin profiles
> > profile: use gz to compress profiles
> >
> > .gitignore | 4 +-
> > CMakeLists.txt | 10 +-
> > framework/profile.py | 203 +-
> > framework/test/glsl_parser_test.py | 60 +-
> > framework/test/piglit_test.py | 68 +-
> > framework/test/shader_test.py | 131 +-
> > tests/CMakeLists.no_api.txt | 89 +-
> > tests/CMakeLists.txt | 2 +-
> > tests/all.meta.xml | 6 +-
> > tests/all.py | 4999 +--------------
> > tests/cl.py | 71 +-
> > tests/cpu.meta.xml | 4 +-
> > tests/cpu.py | 31 +-
> > tests/find_static_tests.py | 74 +-
> > tests/glslparser.py | 63 +-
> > tests/gpu.meta.xml | 5 +-
> > tests/gpu.py | 18 +-
> > tests/llvmpipe.meta.xml | 5 +-
> > tests/llvmpipe.py | 33 +-
> > tests/llvmpipe_gl.py | 33 +-
> > tests/no_error.py | 17 +-
> > tests/opencl_foreign.py | 34 +-
> > tests/opengl.py | 4837 ++++++++++++++-
> > tests/py_modules/constants.py | 8 +-
> > tests/quick.meta.xml | 6 +-
> > tests/quick.py | 97 +-
> > tests/quick_cl.meta.xml | 5 +-
> > tests/quick_cl.py | 37 +-
> > tests/quick_gl.py | 78 +-
> > tests/quick_shader.py | 53 +-
> > tests/serializer.py | 151 +-
> > tests/shader.py | 71 +-
> > unittests/framework/test/test_glsl_parser_test.py | 61 +-
> > unittests/framework/test/test_shader_test.py | 37 +-
> > unittests/framework/test_profile.py | 5 +-
> > 35 files changed, 6009 insertions(+), 5397 deletions(-)
> > create mode 100644 tests/CMakeLists.no_api.txt
> > create mode 100644 tests/all.meta.xml
> > delete mode 100644 tests/all.py
> > create mode 100644 tests/cpu.meta.xml
> > delete mode 100644 tests/cpu.py
> > create mode 100644 tests/find_static_tests.py
> > create mode 100644 tests/gpu.meta.xml
> > delete mode 100644 tests/gpu.py
> > create mode 100644 tests/llvmpipe.meta.xml
> > delete mode 100644 tests/llvmpipe.py
> > create mode 100644 tests/llvmpipe_gl.py
> > create mode 100644 tests/opencl_foreign.py
> > create mode 100644 tests/opengl.py
> > create mode 100644 tests/quick.meta.xml
> > delete mode 100644 tests/quick.py
> > create mode 100644 tests/quick_cl.meta.xml
> > delete mode 100644 tests/quick_cl.py
> > create mode 100644 tests/quick_gl.py
> > create mode 100644 tests/quick_shader.py
> > create mode 100644 tests/serializer.py
> >
> > base-commit: 385fbbe2e73bf4f65f425101e8fa890ae5b52649
> > --
> > git-series 0.9.1
> > _______________________________________________
> > Piglit mailing list
> > Piglit at lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/piglit
>
> Does anyone have interest in reviewing or testing this?
>
> Dylan
> _______________________________________________
> Piglit mailing list
> Piglit at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/piglit
More information about the Piglit
mailing list