[igt-dev] [PATCH i-g-t] tests: Add amdgpu test suite

[Rodrigo Siqueira Jordao]

On 2021-10-11 4:44 a.m., Martin Roukala (néé Peres) wrote:
> 
> 
> On 08/10/2021 22:02, Rodrigo Siqueira wrote:
>> Add various test suites relevant for the amdgpu driver.
>>
>> Cc: Harry Wentland <harry.wentland at amd.com>
>> Cc: Nicholas Choi <Nicholas.Choi at amd.com>
>> Cc: Rodrigo Siqueira <Rodrigo.Siqueira at amd.com>
>> Cc: Sun peng (Leo) Li <sunpeng.li at amd.com>
>> Cc: Alexander Deucher <alexander.deucher at amd.com>
>> Cc: Martin Roukala <martin.roukala at mupuf.org>
>> Cc: Hayden Goodfellow <hayden.goodfellow at amd.com>
>> Cc: Simon Ser <contact at emersion.fr>
>> Cc: Mark Yacoub <markyacoub at google.com>
>>
>> Signed-off-by: Rodrigo Siqueira <Rodrigo.Siqueira at amd.com>
>> ---
>>   tests/amdgpu-ci/README                 |  31 +++++
>>   tests/amdgpu-ci/fast-feedback.testlist | 135 +++++++++++++++++++
>>   tests/amdgpu-ci/full-feedback.testlist | 173 +++++++++++++++++++++++++
>>   3 files changed, 339 insertions(+)
>>   create mode 100644 tests/amdgpu-ci/README
>>   create mode 100644 tests/amdgpu-ci/fast-feedback.testlist
>>   create mode 100644 tests/amdgpu-ci/full-feedback.testlist
>>
>> diff --git a/tests/amdgpu-ci/README b/tests/amdgpu-ci/README
>> new file mode 100644
>> index 00000000..bd34245c
>> --- /dev/null
>> +++ b/tests/amdgpu-ci/README
>> @@ -0,0 +1,31 @@
>> +This directory contains test lists that are used by AMD's CI. The
>> +files are passed to piglit with the --test-list parameter directly.
>> +
>> +The test lists are contained in the IGT repository for several
>> +reasons:
>> +
>> +- The lists stay synchronized with the IGT codebase.
>> +- Public availability. Kernel developers can see what tests are run,
>> +  and can see what changes are done to the set, when, and why.
> 
> Thanks for doing this!
> 
>> +
>> +Changing the test lists should only happen with approval from:
>> +- Harry Wentland <harry.wentland at amd.com>
>> +- Nicholas Choi <Nicholas.Choi at amd.com>
>> +- Rodrigo Siqueira <Rodrigo.Siqueira at amd.com>
>> +- Sun peng (Leo) Li <sunpeng.li at amd.com>
>> +
>> +======================
>> +fast-feedback.testlist
>> +======================
>> +
>> +Fast-feedback contains tests that roughly tests a wide coverage of 
>> features in
>> +a short time. We use this list for presubmission validation. Results 
>> of a
>> +fast-feedback test round should only be considered to mean that the 
>> kernel is
>> +not obviously broken.
> 
> It would be good to have a target execution time for this list. ~10 
> minutes and 12 minutes in the worst case has been working for Intel, but 
> what would you like to use?

Good point,

We already use this testlist in our pre-submission patches.
Nicholas Choi and Hayden, from your experience, what is a good execution 
time for this list?

>> +
>> +======================
>> +full-feedback.testlist
>> +======================
>> +
>> +This is an extensive set of tests that takes a long time to complete 
>> in which
>> +we usually use as post-submission.
> 
>  From experience, I would specify the full list as a opt-out, rather 
> than opt-in. Otherwise, you will soon realize that it takes a lot of 
> time and energy to maintain the list to always add new tests there.
> 
> By opting out of i915/gem/nouveau/udl/... tests, you will at least get 
> every new KMS test by default.
> 
> Is that making sense?

Hmmm... Could you elaborate a little bit more on how I can use the 
opt-out behavior?

Also, I suppose that opting out will enable all tests, which means that 
we will have a lot of failures in this report for AMD, right? We still 
have a lot of IGT tests that only pass on Intel hardware (we are trying 
to fix that), and I'm wondering if it really makes sense to enable 
everything for AMD right now due to this limitation. Additionally, we 
use this full-feeback list in our nightly build to identify regressions 
in our amd-staging-drm-next; that's why we usually don't want to run 
tests that fail in that.

Thanks
Siqueira

> Martin