[Intel-gfx] Making IGT runnable by CI and developers

[Daniel Vetter]

On Mon, Jul 24, 2017 at 09:21:39AM +0100, Tvrtko Ursulin wrote:
> 
> On 21/07/2017 16:52, Daniel Vetter wrote:
> > On Fri, Jul 21, 2017 at 5:13 PM, Jani Nikula
> > <jani.nikula at linux.intel.com> wrote:
> 
> [snip]
> 
> > > I agree the goal should be to run all tests by default. And this means
> > > we should start being more critical of the tests we add.
> > > 
> > > For stress tests I would like to look more into splitting up the tests
> > > in a way that you could run one iteration fast (as part of default), and
> > > repeat the tests for more stress and coverage. I don't know how feasible
> > > this is, and if it requires carrying over state from one iteration to
> > > other, but I like the goal of running also some of this by default. This
> > > would better catch silly bugs in tests too. (We discussed this offline
> > > with Martin and Tomi.)
> > 
> > I think right now, and for the near future (up to at least a year) the
> > only time we'll run stress tests if developers need nastier testcases
> > to help reproduce a bug locally. We simply don't have neither the CI
> > nor the QA resources to run these tests. If we're making really great
> > progress on overall quality and CI infrastructure (that needs budget
> > we don't have right now) and pre-merge testing we might be able to
> > start looking into running stress tests in CI or QA.
> > 
> > I think a good example is kms_frontbuffer_tracking --show-hidden,
> > which Paulo used to debug issues on his own machine.
> 
> Just on this particular point - to make this facility generic would be a
> flavour of test tagging, so pretty much the same high level effect as the
> RFC I sent. I am not pushing it (again), just saying it would be effectively
> the same approach/effort, only that tags are more generic/flexible.

The one difference would be it's binary, as in it only differentiates
between real regression tests everyone should care about vs. tools and
stuff that is only useful for the feature developer. It's _not_ tagging of
everything (which I fully agree with Jani is not going to work if we put
it into the source files).
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch