[Intel-gfx] [PATCH i-g-t 2/3] Unify handling of slow/combinatorial tests

[Daniel Vetter]

On Tue, Oct 27, 2015 at 08:47:28AM +0200, David Weinehall wrote:
> On Mon, Oct 26, 2015 at 03:59:24PM -0200, Paulo Zanoni wrote:
> > 2015-10-26 15:30 GMT-02:00 David Weinehall <david.weinehall at linux.intel.com>:
> > > On Mon, Oct 26, 2015 at 02:44:18PM -0200, Paulo Zanoni wrote:
> > >> 2015-10-26 12:59 GMT-02:00 David Weinehall <david.weinehall at linux.intel.com>:
> > >> > On Fri, Oct 23, 2015 at 11:50:46AM -0200, Paulo Zanoni wrote:
> > >> >
> > >> > [snip]
> > >> >
> > >> >> It's not clear to me, please clarify: now the tests that were
> > >> >> previously completely hidden will be listed in --list-subtests and
> > >> >> will be shown as skipped during normal runs?
> > >> >
> > >> > Yes.  Daniel and I discussed this and he thought listing all test
> > >> > cases, even the slow ones, would not be an issue, since QA should
> > >> > be running the default set not the full list
> > >> > (and for that matter, shouldn't QA know what they are doing too? :P).
> > >>
> > >> If that's the case, I really think your patch should not touch
> > >> kms_frontbuffer_tracking.c. The hidden subtests should not appear on
> > >> the list. People shouldn't even have to ask themselves why they are
> > >> getting 800 skips from a single testcase. Those are only for debugging
> > >> purposes.
> > >
> > > Fair enough.  I'll try to come up with a resonable way to exclude them
> > > from the list in a generic manner.  Because that's the whole point of
> > > this exercise -- to standardise this rather than have every test case
> > > implement its own method of choosing whether or not to run all tests.
> > 
> > Maybe instead of marking these tests as SKIP we could use some other
> > flag. That would avoid the confusion between "skipped because some
> > condition was not match but the test is useful" vs "skipped because
> > the test is unnecessary".
> 
> I'd prefer a method that wouldn't require patching piglit.

The entire "why was this skipped" question is currently unsolved, since
there's also "skipped because old kernel" and "skipped because wrong
platform" and "skipped because not enough ram" and "skipped because wrong
outputs connected" and "skipped because ...". In short, it's a much bigger
problem imo.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch