[Intel-gfx] [PATCH i-g-t 5/5] tests/kms_test_only: Validate TEST_ONLY correctness against full atomic commit

[Maarten Lankhorst]

Op 23-01-17 om 14:26 schreef Mika Kahola:
> On Thu, 2017-01-12 at 12:37 +0100, Maarten Lankhorst wrote:
>> Hey,
>>
>> Op 12-01-17 om 11:28 schreef Mika Kahola:
>>> On Thu, 2017-01-12 at 11:11 +0100, Maarten Lankhorst wrote:
>>>> Op 30-12-16 om 13:00 schreef Mika Kahola:
>>>>> This test case adds TEST_ONLY flag to the following test cases
>>>>> to
>>>>> test
>>>>> atomic commit correctness.
>>>>>
>>>>>  - kms_plane_multiple
>>>>>  - kms_atomic_transitions
>>>>>  - kms_plane_scaling
>>>>>  - kms_rotation_crc
>>>>>
>>>>> The test randomly selects one of the above test cases and tests
>>>>> atomic
>>>>> commit. If the test fails with TEST_ONLY flag the real deal
>>>>> atomic
>>>>> commit
>>>>> is executed and the outcome is verified.
>>>>>
>>>>> The test runs by default for 64 iterations.
>>>>>
>>>>> Signed-off-by: Mika Kahola <mika.kahola at intel.com>
>>>>> ---
>>>>>  tests/Makefile.sources |   1 +
>>>>>  tests/kms_test_only.c  | 455
>>>>> +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>  2 files changed, 456 insertions(+)
>>>>>  create mode 100644 tests/kms_test_only.c
>>>> Hey,
>>>>
>>>> What does this test add?
>>> This is for VIZ-6956. The idea here is to test atomic commits with
>>> TEST_ONLY first, and it that fails we try to commit in real. 
>>>> Any test should pass with PASS. This test retrying failed test-
>>>> only
>>>> tests won't add much there. No test should ever fail.
>>> That's my thought also. You could argue whether or not this test
>>> adds
>>> any value. In my mind, this test only tests if TEST_ONLY flag is
>>> behaving ok :)
>> But you can already compare test passes vs test fails. There's no
>> need for a separate test because any test failing should already be
>> investigated.
>>
>> I don't think a test that tests another test is useful, we already
>> have infrastructure for that. :-)
> So what should we do about this test? Would it be enough if we just add
> these TEST_ONLY flag tests to these igt tests and forget this one that
> just tests if the test was successful or not?
Yes. But we should also probably determine what we want to test.

Easiest might just be to do each commit twice, once with TEST_ONLY, followed by the same commit done normally.

Maybe we could create a test mode for this in igt core that can be toggled, that performs any atomic commit twice?

Plugging that in would be a whole lot less invasive, only adding a single line to each changed test.

~Maarten