[Intel-gfx] The whole round of i-g-t testing cost too long running time

[He, Shuang]

From: Vetter, Daniel
Sent: Wednesday, April 16, 2014 1:18 AM
To: Yang, Guang A; Barnes, Jesse; Widawsky, Benjamin; Wood, Thomas; Jin, Gordon; OTC GFX QA Extended; intel-gfx at lists.freedesktop.org; Parenteau, Paul A; Nikkanen, Kimmo
Subject: Re: The whole round of i-g-t testing cost too long running time

On 15/04/2014 17:46, Yang, Guang A wrote:
Hi all,
I have discussed with Daniel about the running time for each cases before and we set the standard as 10M, if one can't finish after running 10M we will see it as Timeout and report bug on FDO(such as :  Bug 77474<https://bugs.freedesktop.org/show_bug.cgi?id=77474> - [PNV/IVB/HSW]igt/gem_tiled_swapping is slow and Bug 77475<https://bugs.freedesktop.org/show_bug.cgi?id=77475> - [PNV/IVB/HSW]igt//kms_pipe_crc_basic/read-crc-pipe-A is slow)
Now the true status is that i-g-t have more than 650+ subcases, running a whole round of testing will cost such a long time on QA side(beside that Timeout cases), QA also need to spend more time to analysis the result changing on each platforms.
You can find an example with this page: http://tinderbox.sh.intel.com/PRTS_UI/prtsresult.php?task_id=2778 for how long one testing round cost.
With the table of subtask:10831 on the page which for i-g-t test cases on BDW. Testing start at 19:16 PM and finished at 03:25 AM the next day, cost about 8 hours to run 638 test cases.
Each cases finished less than 10M as we expect, but the full time it too large, especially the BDW is the powerful machine on our side, ILK or PNV may take more than 10 hours. We not only run i-g-t but also need to test the piglit/performance/media which already need time.
Do we have any solutions to reduce the running time for whole i-g-t? it's a pressing problem for QA after seeing the i-g-t case count enhance from 50 ->600+.
Ok there are a few cases where we can indeed make tests faster, but it will be work for us. And that won't really speed up much since we're adding piles more testcases at a pretty quick rate. And many of these new testcases are CRC based, so inheritely take some time to run.
[He, Shuang] OK, so it takes at least n/60 in usual case to have result detected plus additional execution time, depending on how many rounds of testing. We will be absolutely happy to see more tests coming that is useful

So I think longer-term we simply need to throw more machines at the problem and run testcases in parallel on identical machines.
[He, Shuang] This would be the perfect way to go if all tests are really feasible to take long time to run. If we get more identical test machines, then problem solved

Wrt analyzing issues I think the right approach for moving forward is:
a) switch to piglit to run tests, not just enumerate them. This will allow QA and developers to share testcase analysis.
[He, Shuang] Yes, though this could not actually accelerate the test. We could directly wrap over piglit to run testing (have other control process to monitor and collecting test results)

b) add automated analysis for time-consuming and error prone cases like dmesg warnings and backtraces. Thomas&I have just discussed a few ideas in this are in our 1:1 today.

Reducing the set of igt tests we run is imo pointless: The goal of igt is to hit corner-cases, arbitrarily selecting which kinds of corner-cases we test just means that we have a nice illusion about our test coverage.
[He, Shuang] I don't think select a subset of test cases to run is pointless. It's a trade-off between speed and correctness. For our nightly testing it's not so useful to run only a small set of testing. But for fast sanity testing, it should be easier, which is supposed to catch regression in major/critical functionality (So other developers and QA could continue their work).


Adding more people to the discussion.

Cheers, Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/intel-gfx/attachments/20140415/8df9d8b6/attachment.html>