[igt-dev] [PATCH i-g-t v3 1/2] tests/intel-ci: Add basic PSR2 tests to fast feedback test list

[Daniel Vetter]

On Wed, Jan 23, 2019 at 09:17:17AM -0800, Rodrigo Vivi wrote:
> On Wed, Jan 23, 2019 at 05:51:11PM +0100, Daniel Vetter wrote:
> > On Wed, Jan 23, 2019 at 5:45 PM Rodrigo Vivi <rodrigo.vivi at intel.com> wrote:
> > >
> > > On Wed, Jan 23, 2019 at 01:07:32PM +0100, Daniel Vetter wrote:
> > > > On Wed, Jan 23, 2019 at 01:37:19PM +0200, Petri Latvala wrote:
> > > > > On Tue, Jan 22, 2019 at 05:09:49PM -0800, José Roberto de Souza wrote:
> > > > > > Lets run the same PSR1 basic tests for PSR2 to caught PSR2
> > > > > > regressions faster.
> > > > > >
> > > > > > Cc: Rodrigo Vivi <rodrigo.vivi at intel.com>
> > > > > > Cc: Dhinakaran Pandiyan <dhinakaran.pandiyan at intel.com>
> > > > > > Signed-off-by: José Roberto de Souza <jose.souza at intel.com>
> > > > > > ---
> > > > > >  tests/intel-ci/fast-feedback.testlist | 4 ++++
> > > > > >  1 file changed, 4 insertions(+)
> > > > > >
> > > > > > diff --git a/tests/intel-ci/fast-feedback.testlist b/tests/intel-ci/fast-feedback.testlist
> > > > > > index da3c4c8e..e48cb8a5 100644
> > > > > > --- a/tests/intel-ci/fast-feedback.testlist
> > > > > > +++ b/tests/intel-ci/fast-feedback.testlist
> > > > > > @@ -227,6 +227,10 @@ igt at kms_psr@primary_page_flip
> > > > > >  igt at kms_psr@cursor_plane_move
> > > > > >  igt at kms_psr@sprite_plane_onoff
> > > > > >  igt at kms_psr@primary_mmap_gtt
> > > > > > +igt at kms_psr@psr2_primary_page_flip
> > > > > > +igt at kms_psr@psr2_cursor_plane_move
> > > > > > +igt at kms_psr@psr2_sprite_plane_onoff
> > > > > > +igt at kms_psr@psr2_primary_mmap_gtt
> > > > >
> > > > >
> > > > > The BAT results mail said success because these are new tests, but do
> > > > > note that they failed. They must pass to get onto the BAT list.
> > > >
> > > > Also, adding all kinds of tests to BAT to validate features doesn't scale.
> > > > We need some way to run these tests on specific machines as part of the
> > > > follow-up shard runs ... Otherwise we're stuck with a huge pressure to add
> > > > all kinds of super-important-feature-right-now things to BAT.
> > >
> > > I understand and I agree with your point. But on this very specific case
> > > no shard have PSR1 or PSR2 panels.
> > 
> > Yeah. Same way that no shard has:
> > -mst
> > -hdcp
> > -dsi
> > -4k
> > - ...
> 
> "coincidentally" all display related :-)
> 
> > 
> > The list is very long. Everyone wants their feature to be an
> > exception. Everyone's feature only increase test time by "not much".
> 
> Yeap, I understand that everybody will put their feature as important,
> but for me another factor that justify that increase is the "fragile"
> part.
> 
> For me the important + fragile deserves a space even if we have to wait
> minutes more for the result :/
> 
> > 
> > > Also this shouldn't increase the test time much, because machines with PSR1 are
> > > already running the PSR1 tests only, machines without PSR are not running
> > > anything and machines. Only machines with PSR2 panels that are now coming from
> > > no PSR tests to running this few PSR2 tests.
> > 
> > Ok, I guess that ship sailed with the psr1 tests already then.
> 
> besides, I think MST also deserves this "privilege" :)

You misunderstood I think, I'm not saying we shouldn't test this. I'm
saying we shouldn't test this in BAT, but solve this problem for real,
through some dedicated machines that run specific tests as part of shards.
That's the real fix, and the fix that scales, and the fix that will allow
us to test a lot more than just a few BAT tests on a few very select
machines.

And imo as feature owners for this, _you_ folks should be fighting for
this, instead of being ok with squeezing a few tests into BAT. That's not
good enough (aside from that it's inefficient).

I want more testing, not less. So should you :-)

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch