[Intel-gfx] The whole round of i-g-t testing cost too long running time

[Jesse Barnes]

On Tue, 15 Apr 2014 19:17:59 +0200
Daniel Vetter <daniel.vetter at intel.com> wrote:
> Ok there are a few cases where we can indeed make tests faster, but it
> will be work for us. And that won't really speed up much since we're
> adding piles more testcases at a pretty quick rate. And many of these
> new testcases are CRC based, so inheritely take some time to run.

But each test should run very quickly in general; I think we have too
many tests that take much longer than they need to.  Adding some
hooks to the driver via debugfs may let us trigger specific cases
directly rather than trying to induce them through massive threading
and memory pressure for example.

And can you elaborate on the CRC tests?  It doesn't seem like those
should take more than a few frames to verify we're getting what we
expect...

> So I think longer-term we simply need to throw more machines at the
> problem and run testcases in parallel on identical machines.
> 
> Wrt analyzing issues I think the right approach for moving forward is:
> a) switch to piglit to run tests, not just enumerate them. This will
> allow QA and developers to share testcase analysis.
> b) add automated analysis for time-consuming and error prone cases like
> dmesg warnings and backtraces. Thomas&I have just discussed a few ideas
> in this are in our 1:1 today.
> 
> Reducing the set of igt tests we run is imo pointless: The goal of igt
> is to hit corner-cases, arbitrarily selecting which kinds of
> corner-cases we test just means that we have a nice illusion about our
> test coverage.

My goal is still to get full test coverage before patches get
committed, and that means having quick (<1hr) turnaround for testing
from the automated patch test system.  It seems like we'll need to
approach that from all angles: speeding up tests, parallelizing
execution, adding hooks to the driver, etc.

Jesse