[Piglit] Nearly finished: shader_runner running THOUSANDS of tests per process

[Dylan Baker]

Quoting Marek Olšák (2016-07-04 16:39:48)
> On Fri, May 27, 2016 at 7:53 PM, Dylan Baker <dylan at pnwbakers.com> wrote:
> > Quoting Marek Olšák (2016-04-16 15:16:34)
> >> Hi,
> >>
> >> This makes shader_runner very fast. The expected result is 40%
> >> decrease in quick.py running time, or a 12x faster piglit run if you
> >> run shader tests alone.
> >>
> >> Branch:
> >> https://cgit.freedesktop.org/~mareko/piglit/log/?h=shader-runner
> >>
> >> Changes:
> >>
> >> 1) Any number of test files can be specified as command-line
> >> parameters. Those command lines can be insanely long.
> >>
> >> 2) shader_runner can re-create the window & GL context if test
> >> requirements demand different settings when going from one test to
> >> another.
> >>
> >> 3) all.py generates one shader_runner instance per group of tests
> >> (usually one or two directories - tests and generated_tests).
> >> Individual tests are reported as subtests.
> >>
> >> The shader_runner part is done. The python part needs more work.
> >>
> >>
> >> What's missing:
> >>
> >> Handling of crashes. If shader_runner crashes:
> >> - The crash is not shown in piglit results (other tests with subtests
> >> already have the same behavior)
> >> - The remaining tests will not be run.
> >>
> >> The ShaderTest python class has the list of all files and should be
> >> able to catch a crash, check how many test results have been written,
> >> and restart shader_runner with the remaining tests.
> >>
> >> shader_runner prints TEST %i: and then the subtest result. %i is the
> >> i-th file in the list. Python can parse that and re-run shader_runner
> >> with the first %i tests removed. (0..%i-1 -> parse subtest results; %i
> >> -> crash; %i+1.. -> run again)
> >>
> >>
> >> I'm by no means a python expert, so here's an alternative solution (for me):
> >> - Catch crash signals in shader_runner.
> >> - In the single handler, re-run shader_runner with the remaining tests.
> >>
> >> Opinions welcome,
> >>
> >> Marek
> >> _______________________________________________
> >> Piglit mailing list
> >> Piglit at lists.freedesktop.org
> >> https://lists.freedesktop.org/mailman/listinfo/piglit
> >
> > Hey Marek,
> >
> > I'd picked this up and was finishing it up, I have a branch on my github
> > (https://github.com/dcbaker/piglit wip/multi-shader_runner), I'm just
> > trying to make sure we're not duplicating effort.
> 
> Hi,
> 
> What's the current state of this please?
> 
> Do you need shader_runner to be able to recover from crashes or does
> the framework handle them already?
> 
> Marek

Sorry for the late reply, it was a holiday here in the US,

I've set up the framework to recover from them. My assumption was that
this would be more portable than C, since we would probably need windows
specific code and linux specific code.

I was planning to pick this up again today, I've had other things that
needed to be solved first, but basically I'm down to the bug squashing
and patch ordering stage, plus handling a few corner cases (there are a
couple of extensions that apply to GLES and to desktop GL).

Dylan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: signature
URL: <https://lists.freedesktop.org/archives/piglit/attachments/20160706/8a108328/attachment.sig>