[igt-dev] [PATCH i-g-t v3 1/2] tests/intel-ci: Add basic PSR2 tests to fast feedback test list

[Rodrigo Vivi]

On Thu, Jan 24, 2019 at 01:55:41PM +0100, Daniel Vetter wrote:
> On Wed, Jan 23, 2019 at 09:17:17AM -0800, Rodrigo Vivi wrote:
> > On Wed, Jan 23, 2019 at 05:51:11PM +0100, Daniel Vetter wrote:
> > > On Wed, Jan 23, 2019 at 5:45 PM Rodrigo Vivi <rodrigo.vivi at intel.com> wrote:
> > > >
> > > > On Wed, Jan 23, 2019 at 01:07:32PM +0100, Daniel Vetter wrote:
> > > > > On Wed, Jan 23, 2019 at 01:37:19PM +0200, Petri Latvala wrote:
> > > > > > On Tue, Jan 22, 2019 at 05:09:49PM -0800, José Roberto de Souza wrote:
> > > > > > > Lets run the same PSR1 basic tests for PSR2 to caught PSR2
> > > > > > > regressions faster.
> > > > > > >
> > > > > > > Cc: Rodrigo Vivi <rodrigo.vivi at intel.com>
> > > > > > > Cc: Dhinakaran Pandiyan <dhinakaran.pandiyan at intel.com>
> > > > > > > Signed-off-by: José Roberto de Souza <jose.souza at intel.com>
> > > > > > > ---
> > > > > > >  tests/intel-ci/fast-feedback.testlist | 4 ++++
> > > > > > >  1 file changed, 4 insertions(+)
> > > > > > >
> > > > > > > diff --git a/tests/intel-ci/fast-feedback.testlist b/tests/intel-ci/fast-feedback.testlist
> > > > > > > index da3c4c8e..e48cb8a5 100644
> > > > > > > --- a/tests/intel-ci/fast-feedback.testlist
> > > > > > > +++ b/tests/intel-ci/fast-feedback.testlist
> > > > > > > @@ -227,6 +227,10 @@ igt at kms_psr@primary_page_flip
> > > > > > >  igt at kms_psr@cursor_plane_move
> > > > > > >  igt at kms_psr@sprite_plane_onoff
> > > > > > >  igt at kms_psr@primary_mmap_gtt
> > > > > > > +igt at kms_psr@psr2_primary_page_flip
> > > > > > > +igt at kms_psr@psr2_cursor_plane_move
> > > > > > > +igt at kms_psr@psr2_sprite_plane_onoff
> > > > > > > +igt at kms_psr@psr2_primary_mmap_gtt
> > > > > >
> > > > > >
> > > > > > The BAT results mail said success because these are new tests, but do
> > > > > > note that they failed. They must pass to get onto the BAT list.
> > > > >
> > > > > Also, adding all kinds of tests to BAT to validate features doesn't scale.
> > > > > We need some way to run these tests on specific machines as part of the
> > > > > follow-up shard runs ... Otherwise we're stuck with a huge pressure to add
> > > > > all kinds of super-important-feature-right-now things to BAT.
> > > >
> > > > I understand and I agree with your point. But on this very specific case
> > > > no shard have PSR1 or PSR2 panels.
> > > 
> > > Yeah. Same way that no shard has:
> > > -mst
> > > -hdcp
> > > -dsi
> > > -4k
> > > - ...
> > 
> > "coincidentally" all display related :-)
> > 
> > > 
> > > The list is very long. Everyone wants their feature to be an
> > > exception. Everyone's feature only increase test time by "not much".
> > 
> > Yeap, I understand that everybody will put their feature as important,
> > but for me another factor that justify that increase is the "fragile"
> > part.
> > 
> > For me the important + fragile deserves a space even if we have to wait
> > minutes more for the result :/
> > 
> > > 
> > > > Also this shouldn't increase the test time much, because machines with PSR1 are
> > > > already running the PSR1 tests only, machines without PSR are not running
> > > > anything and machines. Only machines with PSR2 panels that are now coming from
> > > > no PSR tests to running this few PSR2 tests.
> > > 
> > > Ok, I guess that ship sailed with the psr1 tests already then.
> > 
> > besides, I think MST also deserves this "privilege" :)
> 
> You misunderstood I think, I'm not saying we shouldn't test this. I'm
> saying we shouldn't test this in BAT, but solve this problem for real,
> through some dedicated machines that run specific tests as part of shards.
> That's the real fix, and the fix that scales, and the fix that will allow
> us to test a lot more than just a few BAT tests on a few very select
> machines.

Oh! I see now... That's indeed a very smarter way of scaling this.

And maybe not necessarily "shard" machines and not necessarily running all IGT.
And maybe some specific feature-machine.testlist that is part of the
second round of CI-IGT...

Martin? :$

> 
> And imo as feature owners for this, _you_ folks should be fighting for
> this, instead of being ok with squeezing a few tests into BAT. That's not
> good enough (aside from that it's inefficient).
> 
> I want more testing, not less. So should you :-)
> 
> Cheers, Daniel
> -- 
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch
> _______________________________________________
> igt-dev mailing list
> igt-dev at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/igt-dev