[Intel-gfx] [PATCH i-g-t 5/5] tests/kms_test_only: Validate TEST_ONLY correctness against full atomic commit

[Mika Kahola]

On Thu, 2017-01-12 at 12:37 +0100, Maarten Lankhorst wrote:
> Hey,
> 
> Op 12-01-17 om 11:28 schreef Mika Kahola:
> > 
> > On Thu, 2017-01-12 at 11:11 +0100, Maarten Lankhorst wrote:
> > > 
> > > Op 30-12-16 om 13:00 schreef Mika Kahola:
> > > > 
> > > > This test case adds TEST_ONLY flag to the following test cases
> > > > to
> > > > test
> > > > atomic commit correctness.
> > > > 
> > > >  - kms_plane_multiple
> > > >  - kms_atomic_transitions
> > > >  - kms_plane_scaling
> > > >  - kms_rotation_crc
> > > > 
> > > > The test randomly selects one of the above test cases and tests
> > > > atomic
> > > > commit. If the test fails with TEST_ONLY flag the real deal
> > > > atomic
> > > > commit
> > > > is executed and the outcome is verified.
> > > > 
> > > > The test runs by default for 64 iterations.
> > > > 
> > > > Signed-off-by: Mika Kahola <mika.kahola at intel.com>
> > > > ---
> > > >  tests/Makefile.sources |   1 +
> > > >  tests/kms_test_only.c  | 455
> > > > +++++++++++++++++++++++++++++++++++++++++++++++++
> > > >  2 files changed, 456 insertions(+)
> > > >  create mode 100644 tests/kms_test_only.c
> > > Hey,
> > > 
> > > What does this test add?
> > This is for VIZ-6956. The idea here is to test atomic commits with
> > TEST_ONLY first, and it that fails we try to commit in real. 
> > > 
> > > Any test should pass with PASS. This test retrying failed test-
> > > only
> > > tests won't add much there. No test should ever fail.
> > That's my thought also. You could argue whether or not this test
> > adds
> > any value. In my mind, this test only tests if TEST_ONLY flag is
> > behaving ok :)
> But you can already compare test passes vs test fails. There's no
> need for a separate test because any test failing should already be
> investigated.
> 
> I don't think a test that tests another test is useful, we already
> have infrastructure for that. :-)
So what should we do about this test? Would it be enough if we just add
these TEST_ONLY flag tests to these igt tests and forget this one that
just tests if the test was successful or not?

> 
> ~Maarten
-- 
Mika Kahola - Intel OTC