[Intel-gfx] [PATCH i-g-t 2/3] Unify handling of slow/combinatorial tests

[David Weinehall]

On Mon, Oct 26, 2015 at 02:44:18PM -0200, Paulo Zanoni wrote:
> 2015-10-26 12:59 GMT-02:00 David Weinehall <david.weinehall at linux.intel.com>:
> > On Fri, Oct 23, 2015 at 11:50:46AM -0200, Paulo Zanoni wrote:
> >
> > [snip]
> >
> >> It's not clear to me, please clarify: now the tests that were
> >> previously completely hidden will be listed in --list-subtests and
> >> will be shown as skipped during normal runs?
> >
> > Yes.  Daniel and I discussed this and he thought listing all test
> > cases, even the slow ones, would not be an issue, since QA should
> > be running the default set not the full list
> > (and for that matter, shouldn't QA know what they are doing too? :P).
> 
> If that's the case, I really think your patch should not touch
> kms_frontbuffer_tracking.c. The hidden subtests should not appear on
> the list. People shouldn't even have to ask themselves why they are
> getting 800 skips from a single testcase. Those are only for debugging
> purposes.

Fair enough.  I'll try to come up with a resonable way to exclude them
from the list in a generic manner.  Because that's the whole point of
this exercise -- to standardise this rather than have every test case
implement its own method of choosing whether or not to run all tests.

> >
> >> For kms_frontbuffer_tracking, hidden tests are supposed to be just for
> >> developers who know what they are doing. I hide them behind a special
> >> command-line switch that's not used by QA because I don't want QA
> >> wasting time running those tests. One third of the
> >> kms_frontbuffer_tracking hidden tests only serve the purpose of
> >> checking whether there's a bug in kms_frontbuffer_track itself or not.
> >> For some other hidden tests, they are there just to help better debug
> >> in case some other non-hidden tests fail. Some other hidden tests are
> >> 100% useless and superfluous.
> >
> > Shouldn't 100% useless and superfluous tests be excised completely?
> 
> The change would be from "if (case && hidden) continue;" to "if (case)
> continue;". But that's not the focus. There are still tests that are
> useful for debugging but useless for QA.

It's not the focus of my change, no. But if there are tests that are
useless and/or superfluous, then they should be dropped.  Note that
I'm not suggesting that all non-default tests be dropped, just that
if there indeed are tests that don't make sense, they shouldn't be
in the test case in the first place.

> >
> >> QA should only run the non-hidden tests.
> >
> > Which is the default behaviour, AFAICT.
> 
> Then why do you want to expose those tests that you're not even
> planning to run??

To allow developers to see the options they have?

> You're kinda implying that QA - or someone else -
> will run those tests at some point, and I say that, for
> kms_frontbuffer_tracking, that's a waste of time. Maybe this is the
> case for the other tests you're touching, but not here.

No, I'm not implying that -- you're putting those words in my mouth.

Anyway, the choice to expose all cases, not just those run without
specifying --all, was a suggestion by Daniel -- you'll have to prod him
to hear what his reasoning was.


Regards, David