[Piglit] Waffle and piglit-dispatch plans

[Kenneth Graunke]

On 05/30/2012 07:51 AM, Pauli Nieminen wrote:
> On Tue, May 29, 2012 at 03:10:53PM -0700, Kenneth Graunke wrote:
>> On 05/29/2012 01:47 PM, Pauli Nieminen wrote:
>>> Wouldn't that makes it simpler to have same tests set in all
>>> platforms and GL flavors? Tests automatically skip when given
>>> environment doesn't match what test requires. Fork-exec-skip should be
>>> fast enough not to affect total runtime much.
>>
>> It actually does take a while.  Running all the 3.0+/1.30+ tests on
>> pre-Sandybridge skips quite a few tests already...and more all the time.
>>
> 
> Right. I forgot skiping requires setting up GL context that takes
> quite some time.
> 
>> Plus, we'd eventually like to include FBConfigs/visuals in the list of
>> requirements (i.e. RGBA, at least N bits of depth, stencil), and that
>> will create a -lot- of combinations.
>>
>> The idea is to improve the runner infrastructure to select reasonable
>> combinations of tests rather than always running them all.
> 
> That would actually make single test runner binary that is optionally
> controlled using stdin to run multiple tests would complete tests a
> lot faster. 
> 
> Quickly looking  amsparsetest with kernelshark. Too bad userspace is
> compiled with -fomit-framepointer so no userspace callchains
> available.
> 0ms     asmparsetest starts
>         First hald of this 6ms goes with frequent syscalls (shared
>         library loading?) while second half has relatively few
>         mprotect calls (Resolve symbols?).
> 5.9ms   connect to Xorg
>         Lots of round trips to X
> 8.3ms   connection established (up to DRI2Connect?)
> 9.1ms   Looks like here we have i965 driver loaded
>         Userspace processing follows
> 11.9ms  Some X communications (possible DRI2Authenenticate)
>         Mostly userspace processing but a few files opened too
> 16.8ms  Create window
> 16.9ms  MakeCurrent (DRI2GetBuffers blocking X for 0.6ms because WM
>         comunication)
> 19.4ms  Test skips after noticing required extension is missing (from
>         shader source).
> 
> Most of that time is either shared library loading (maybe 7ms) and
> context and window setup (maybe 10ms).

Cool!  I've never seen a breakdown of the time like this.  That is a
fair amount of overhead...

> I suppose we could eliminate large part of that with single test
> runner binary that executes multiple tests loaded from shared
> libraries using dlopen. The test runner could pass test cases and
> parameters using stdin to keep test run control in python.
> 
> But I don't know if that is something we want to have.
> 
> Pauli

Yeah, that would be faster.  But I do like Piglit's "a test is just an
executable that prints a result in a format we understand" methodology.
 I really like how things are in a separate process - it protects us
against crashes/assertion failures, bugs where we fail to restore GL
state properly, memory leaks...

It sounds like people want to try putting multiple tests in a single
run.  With a SIGSEGV handler, we can probably get away with that.  I
wouldn't mind seeing someone try it, at least for the
{asm,glsl}parsertests.  Of course, having just a subset in a single
executable makes concurrency more complicated...you can't just schedule
a series of binaries across cores, but have to take into account how
many tests that binary would run.  Maybe run N glslparsertests each
doing 1/N of the tests...

--Ken