[Intel-gfx] Making IGT runnable by CI and developers

[Tvrtko Ursulin]

On 21/07/2017 16:45, Daniel Vetter wrote:
> On Fri, Jul 21, 2017 at 12:56 PM, Tvrtko Ursulin
> <tvrtko.ursulin at linux.intel.com> wrote:
>>
>> On 20/07/2017 17:23, Martin Peres wrote:
>>>
>>> Hi everyone,
>>>
>>> As some of you may already know, we have made great strides in making our
>>> CI system usable, especially in the last 6 months when everything started
>>> clicking together.
>>>
>>> The CI team is no longer overwhelmed with fires and bug reports, so we
>>> started working on increasing the coverage from just fast-feedback, to a
>>> bigger set of IGT tests.
>>>
>>> As some of you may know, running IGT has been a challenge that few manage
>>> to overcome. Not only is the execution time counted in machine months, but
>>> it can also lead to disk corruption, which does not encourage developers to
>>> run it either. One test takes 21 days, on its own, and it is a subset of
>>> another test which we never ran for obvious reasons.
>>>
>>> I would thus like to get the CI team and developers to work together to
>>> decrease sharply the execution time of IGT, and get these tests run multiple
>>> times per day!
>>>
>>> There are three usages that the CI team envision (up for debate):
>>>    - Basic acceptance testing: Meant for developers and CI to check quickly
>>> if a patch series is not completely breaking the world (< 10 minutes,
>>> timeout per test of 30s)
>>>    - Full run: Meant to be ran overnight by developers and users (< 6
>>> hours)
>>
>>
>> We could start by splitting this budget to logical components/teams.
>>
>> So far we have been talking about GEM and KMS, but I was just thinking that
>> we may want to have a separate units on this level of likes of power
>> management, DRM (core), external stuff like sw fences? TBD I guess.
>>
>> Assuming GEM/KMS split only, fair thing seems to be split the time budget
>> 50-50 and let the respective teams start working.
> 
> Yes, KMS is also not perfect, but there it's maybe a factor of 2x that
> it's taking too long. GEM is 50x or worse. Also note KMS includes
> everything, so core drm, PM tests. 2x is something can be fixed as we
> go, which is good, since it means we should be able to pre-merge test
> any changes to igt before pushing. GEM is not even close.
> 
>> I assume this is x hours on the slowest machine?
>>
>> Teams would also need easy access to up-to-date test run times.
> 
> Right now you can't have that for GEM, because it takes 24d. That
> means 1 run of GEM takes away 50 runs of everything else (need to
> check, it might be worse). There's simply no way we can even hand out
> that data without blocking pre-merge CI for everyone else.
> 
> We might be able to schedule the occasional manual run over the w/e,
> but that's about it.

I did not explain well here what I was thinking about by access to 
up-to-date runtime. I assumed we would start from a cut down list, the 
one which fits in the time budget.

As Martin and me chatted on Friday, I would be completely fine with the 
CI team just picking a list of GEM tests which fits, and then the GEM 
team responsibility is to add, remove and improve tests until this time 
is used in the most optimal way.

This was we would be getting daily test run time updates.

We also talked about the idea to set up an IGT trybot, where we could 
send test changes, followed by a testlist updates, and so see the 
specific test runtimes across the platforms.

Once that looks ok, we could submit a patch to the real test list and so 
keep iterating until the above goal is reached.

Regards,

Tvrtko