[igt-dev] [PATCH i-g-t v3 1/2] tests/kms_flip: Set duration for subtest from command line

[Maarten Lankhorst]

Op 08-11-18 om 13:45 schreef Kahola, Mika:
> On Thu, 2018-11-08 at 12:47 +0100, Maarten Lankhorst wrote:
>> Op 09-08-18 om 12:12 schreef Mika Kahola:
>>> To reduce the execution time of kms_flip test on CI, let's move
>>> subtest
>>> duration parameter as command line option. The default subtest
>>> duration
>>> is 3 seconds for test that require jitter computation and for the
>>> rest
>>> of the subtests are run only once.
>>>
>>> v2: Run each subtest only once (default action)
>>> v3: Reduces default timeout for tests that require jitter
>>> computation (Ville)
>>>
>>> Signed-off-by: Mika Kahola <mika.kahola at intel.com>
>> I think this is the wrong approach. What we really want to do is kill
>> tests from kms_flip
>> instead and move them to separate places.
>>
>> Killing off all the interruptible tests would save 50% of the time.
>> So all we have to do is
>> making sure that we have tests that test the missing ioctl's in in
>> kms_atomic_interruptible,
>> and we would save 50% of the time.
> kms_atomic_interruptible would probably be more logical place for
> interrupt tests. I agree that it would reduce the execution time of the
> kms_flip test but in the end of the day we still need to run those
> interruptible tests and therefore we would end up increasing the
> execution time of the kms_atomic_interruptible test.
>
> Do you feel that we could have overall reduction in test execution if
> we move these interruptible tests out from kms_flip? Maybe kms_flip
> readability would improve and therefore worth the effort?
The kms_flip ones just double the execution time of the tests, and don't result in more coverage.

Testing the various ioctl's in a controlled fashion would result in better coverage and lower execution time.
Just look at how kms_atomic_interruptible is structure, it should be easy to add more tests to it and actually
guarantees that the ioctl being tested is interrupted.

I definitely hope that removing those tests increase readability, because of all the unrelated API test special
cases being removed, and only actual flip related tests remaining.

~Maarten