[Piglit] faster piglit start-up

[Brian Paul]

On 04/12/2017 02:32 PM, Eric Anholt wrote:
> Brian Paul <brianp at vmware.com> writes:
>
>> When we run piglit the all.py script traverses the entire tests/ and
>> generated/tests/ directories looking for *.shader_test, *.vert, *.frag,
>> etc. shaders to run.  In the generated_tests/ directory, there's about
>> 60,000 entries to scan (ls -R generated_tests|wc)
>>
>> This takes some time, especially so with MinGW in a Windows VM.  Between
>> that and other setup overhead it takes about 8 minutes on my system
>> before any tests are run.  Ugh.  Linux is much faster but there's still
>> a start-up lag.
>>
>> I'd like to trim this down.  The prime area I'm looking at is skipping
>> swaths of tests when we know the particular feature is not supported by
>> the GL driver.
>>
>> In framework/driver_classifier.py we run glxinfo to get the GL renderer
>> string, etc.  I'm looking at expanding this to get the driver's GL and
>> GLSL versions, extension list, etc.
>>
>> Then, in all.py when we're scanning the tests directory and we find
>> somthing like "spec/arb_gpu_shader_fp64" which may not be supported by
>> the driver, we can skip that directory.  For drivers which don't support
>> all the latest GL 4.x extensions (like VMware's) we can save a lot of
>> time by skipping subdirectory traversals and building the test list.
>>
>> I'm digging into this now but it'll probably be a few days before I'll
>> have something to review.
>>
>> I'd be interested to hear any other ideas to reduce start-up time.
>
> vc4 is also terrible to test these days due to the explosion of
> generated shader_tests we have -- I've got them hacked out of my builds
> due to FS space used and the minutes it takes to start piglit when
> they're present.
>
> For a solution to the startup time, I wish we would just build our whole
> test list using filesystem walks or whatever at piglit build time,
> instead of executing python to build the test list per piglit run.

I like that idea.

Though, I'd still like a way to skip running all the tests for 
"spec at GL_EXT_foobar" if we know that GL_EXT_foobar isn't supported by 
the driver we're testing.  Or similarly for GL x.y or GLSL x.y.

If we have N GL_EXT_foobar tests, it's kind of crazy to start N 
processes, setup a GL context then return SKIP as soon as 
piglit_require_extension("GL_EXT_foobar") fails.  N is pretty large for 
some extensions nowadays.

I know we can pass a bunch of -x options to piglit-run.py to effectively 
do that, but that's a lot of busywork.

-Brian