[igt-dev] Intel CI blacklist suggestion for tests that are skipped on all shards

[Lofstedt, Marta]

> -----Original Message-----
> From: Jani Nikula [mailto:jani.nikula at linux.intel.com]
> Sent: Wednesday, February 7, 2018 1:06 PM
> To: Lofstedt, Marta <marta.lofstedt at intel.com>; igt-
> dev at lists.freedesktop.org
> Cc: Sarvela, Tomi P <tomi.p.sarvela at intel.com>; Peres, Martin
> <martin.peres at intel.com>
> Subject: RE: [igt-dev] Intel CI blacklist suggestion for tests that are skipped on
> all shards
> 
> On Wed, 07 Feb 2018, "Lofstedt, Marta" <marta.lofstedt at intel.com> wrote:
> >> -----Original Message-----
> >> From: Jani Nikula [mailto:jani.nikula at linux.intel.com]
> >> Sent: Tuesday, February 6, 2018 8:42 PM
> >> To: Lofstedt, Marta <marta.lofstedt at intel.com>; igt-
> >> dev at lists.freedesktop.org
> >> Cc: Sarvela, Tomi P <tomi.p.sarvela at intel.com>; Peres, Martin
> >> <martin.peres at intel.com>
> >> Subject: Re: [igt-dev] Intel CI blacklist suggestion for tests that
> >> are skipped on all shards
> >>
> >> On Tue, 06 Feb 2018, "Lofstedt, Marta" <marta.lofstedt at intel.com>
> wrote:
> >> > After summarizing the skip time for our shards:
> >> > GLK: 428s, APL: 371s, SNB:325s, KBL:283, HSW:182s
> >>
> >> Why does it take so long on any of those platforms to skip the tests?
> >
> > I believe it is the huge amount of skipped tests that is the main
> > culprit, ~1500*0,1 seconds is real time. And the new DRRS test
> > contribute a lot.
> >
> > Also, I summarized some timings of the the kms_frontbuffer_tracking on
> > GLK-shards:
> >
> > Total time = 2153s, note this is more than half of the total GLK-shard
> > runtime.  Skipped time = 238s, this has more than doubled due to new
> > DRRS tests
> 
> > Fail time = 1365s, this is mostly for the PSR tests that have been
> > failing for months, due to unsupported panels being shipped with GLK
> > RVPs. Unfortunately we don't have a full set of supported PSR panels.
> > And if we changed to supported PSR panels I assume that the runtime
> > would only increase further.
> 
> Okay. I'm just wondering about the possibilities of optimizing the skipping in
> the tests themselves, perhaps by bailing out earlier, vs. maintainability of a
> single big blacklist which combines plenty of features across plenty of
> platforms.

VIZ-8825 could do that, but I usually prefer picking low hanging fruit before redefining how CI run tests and how IGT test are structured.

> 
> BR,
> Jani.
> 
> 
> >
> >
> >>
> >> BR,
> >> Jani.
> >>
> >>
> >> --
> >> Jani Nikula, Intel Open Source Technology Center
> 
> --
> Jani Nikula, Intel Open Source Technology Center