[Piglit] Nearly finished: shader_runner running THOUSANDS of tests per process

[Marek Olšák]

On Fri, May 27, 2016 at 7:53 PM, Dylan Baker <dylan at pnwbakers.com> wrote:
> Quoting Marek Olšák (2016-04-16 15:16:34)
>> Hi,
>>
>> This makes shader_runner very fast. The expected result is 40%
>> decrease in quick.py running time, or a 12x faster piglit run if you
>> run shader tests alone.
>>
>> Branch:
>> https://cgit.freedesktop.org/~mareko/piglit/log/?h=shader-runner
>>
>> Changes:
>>
>> 1) Any number of test files can be specified as command-line
>> parameters. Those command lines can be insanely long.
>>
>> 2) shader_runner can re-create the window & GL context if test
>> requirements demand different settings when going from one test to
>> another.
>>
>> 3) all.py generates one shader_runner instance per group of tests
>> (usually one or two directories - tests and generated_tests).
>> Individual tests are reported as subtests.
>>
>> The shader_runner part is done. The python part needs more work.
>>
>>
>> What's missing:
>>
>> Handling of crashes. If shader_runner crashes:
>> - The crash is not shown in piglit results (other tests with subtests
>> already have the same behavior)
>> - The remaining tests will not be run.
>>
>> The ShaderTest python class has the list of all files and should be
>> able to catch a crash, check how many test results have been written,
>> and restart shader_runner with the remaining tests.
>>
>> shader_runner prints TEST %i: and then the subtest result. %i is the
>> i-th file in the list. Python can parse that and re-run shader_runner
>> with the first %i tests removed. (0..%i-1 -> parse subtest results; %i
>> -> crash; %i+1.. -> run again)
>>
>>
>> I'm by no means a python expert, so here's an alternative solution (for me):
>> - Catch crash signals in shader_runner.
>> - In the single handler, re-run shader_runner with the remaining tests.
>>
>> Opinions welcome,
>>
>> Marek
>> _______________________________________________
>> Piglit mailing list
>> Piglit at lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/piglit
>
> Hey Marek,
>
> I'd picked this up and was finishing it up, I have a branch on my github
> (https://github.com/dcbaker/piglit wip/multi-shader_runner), I'm just
> trying to make sure we're not duplicating effort.

Hi,

What's the current state of this please?

Do you need shader_runner to be able to recover from crashes or does
the framework handle them already?

Marek