[Intel-gfx] [PATCH i-g-t v2] tests: Add explicit test lists for CI use.

Thu Oct 6 09:22:54 UTC 2016

Regardless of the list of tests we expect to pass, I think it's better 
to have an actual list rather than relying on the name of the tests (as 
we have been doing or are still doing, correct me if I'm wrong).

Thanks for doing this!

Acked-by: Lionel Landwerlin <lionel.g.landwerlin at intel.com>

On 06/10/16 09:54, Petri Latvala wrote:
> See tests/intel-ci/README for rationale and explanation.
>
> v2: Use the current BAT set for fast-feedback.testlist first
>
> Signed-off-by: Petri Latvala <petri.latvala at intel.com>
> ---
>
> Thanks for the feedback. This commit only contains the current test
> set, extended test suites are left for future discussion.
>
>
>   tests/intel-ci/README                 |  34 +++++
>   tests/intel-ci/fast-feedback.testlist | 248 ++++++++++++++++++++++++++++++++++
>   2 files changed, 282 insertions(+)
>   create mode 100644 tests/intel-ci/README
>   create mode 100644 tests/intel-ci/fast-feedback.testlist
>
> diff --git a/tests/intel-ci/README b/tests/intel-ci/README
> new file mode 100644
> index 0000000..dc7e5c8
> --- /dev/null
> +++ b/tests/intel-ci/README
> @@ -0,0 +1,34 @@
> +This directory contains test lists that are used by Intel's CI. The
> +files are passed to piglit with the --test-list parameter directly.
> +
> +The test lists are contained in the IGT repository for several
> +reasons:
> +
> +- The lists stay synchronized with the IGT codebase.
> +- Public availability. Kernel developers can see what tests are run,
> +  and can see what changes are done to the set, when, and why.
> +- Explicit test lists in general make it possible to implement a new
> +  test without having it run in CI (accidentally or on purpose) before
> +  CI is ready for it, and for CI to stop running a test without
> +  removing the test or changing its name. The previous method of test
> +  selection was based on name patterns.
> +
> +Changing the test lists should only happen with approval from the CI
> +people. If you make a new test that should be tested by CI regularly,
> +ask a maintainer (see MAINTAINERS).
> +
> +======================
> +fast-feedback.testlist
> +======================
> +
> +Fast-feedback is the testlist that is run for Patchwork and Trybot
> +testing. It is also run on various drm-intel-* branches.
> +
> +Time limit: Full run must complete in 15 minutes on all tested
> +platforms.
> +
> +Feature coverage: Should test all working features, with at least 1
> +test per feature.
> +
> +The string "basic" in a test name means the test probably belongs in
> +this list.
> diff --git a/tests/intel-ci/fast-feedback.testlist b/tests/intel-ci/fast-feedback.testlist
> new file mode 100644
> index 0000000..ae94381
> --- /dev/null
> +++ b/tests/intel-ci/fast-feedback.testlist
> @@ -0,0 +1,248 @@
> +igt at core_auth@basic-auth
> +igt at core_prop_blob@basic
> +igt at drv_getparams_basic@basic-eu-total
> +igt at drv_getparams_basic@basic-subslice-total
> +igt at drv_hangman@error-state-basic
> +igt at drv_module_reload_basic
> +igt at gem_basic@bad-close
> +igt at gem_basic@create-close
> +igt at gem_basic@create-fd-close
> +igt at gem_busy@basic-busy-default
> +igt at gem_busy@basic-hang-default
> +igt at gem_close_race@basic-process
> +igt at gem_close_race@basic-threads
> +igt at gem_cpu_reloc@basic
> +igt at gem_cs_tlb@basic-default
> +igt at gem_ctx_basic
> +igt at gem_ctx_create@basic
> +igt at gem_ctx_create@basic-files
> +igt at gem_ctx_exec@basic
> +igt at gem_ctx_param@basic
> +igt at gem_ctx_param@basic-default
> +igt at gem_ctx_switch@basic-default
> +igt at gem_ctx_switch@basic-default-heavy
> +igt at gem_exec_basic@basic-blt
> +igt at gem_exec_basic@basic-bsd
> +igt at gem_exec_basic@basic-bsd1
> +igt at gem_exec_basic@basic-bsd2
> +igt at gem_exec_basic@basic-default
> +igt at gem_exec_basic@basic-render
> +igt at gem_exec_basic@basic-vebox
> +igt at gem_exec_basic@gtt-blt
> +igt at gem_exec_basic@gtt-bsd
> +igt at gem_exec_basic@gtt-bsd1
> +igt at gem_exec_basic@gtt-bsd2
> +igt at gem_exec_basic@gtt-default
> +igt at gem_exec_basic@gtt-render
> +igt at gem_exec_basic@gtt-vebox
> +igt at gem_exec_basic@readonly-blt
> +igt at gem_exec_basic@readonly-bsd
> +igt at gem_exec_basic@readonly-bsd1
> +igt at gem_exec_basic@readonly-bsd2
> +igt at gem_exec_basic@readonly-default
> +igt at gem_exec_basic@readonly-render
> +igt at gem_exec_basic@readonly-vebox
> +igt at gem_exec_create@basic
> +igt at gem_exec_flush@basic-batch-kernel-default-cmd
> +igt at gem_exec_flush@basic-batch-kernel-default-uc
> +igt at gem_exec_flush@basic-batch-kernel-default-wb
> +igt at gem_exec_flush@basic-uc-pro-default
> +igt at gem_exec_flush@basic-uc-prw-default
> +igt at gem_exec_flush@basic-uc-ro-default
> +igt at gem_exec_flush@basic-uc-rw-default
> +igt at gem_exec_flush@basic-uc-set-default
> +igt at gem_exec_flush@basic-wb-pro-default
> +igt at gem_exec_flush@basic-wb-prw-default
> +igt at gem_exec_flush@basic-wb-ro-before-default
> +igt at gem_exec_flush@basic-wb-ro-default
> +igt at gem_exec_flush@basic-wb-rw-before-default
> +igt at gem_exec_flush@basic-wb-rw-default
> +igt at gem_exec_flush@basic-wb-set-default
> +igt at gem_exec_gttfill@basic
> +igt at gem_exec_nop@basic-parallel
> +igt at gem_exec_nop@basic-series
> +igt at gem_exec_parallel@basic
> +igt at gem_exec_parse@basic-allowed
> +igt at gem_exec_parse@basic-rejected
> +igt at gem_exec_store@basic-all
> +igt at gem_exec_store@basic-blt
> +igt at gem_exec_store@basic-bsd
> +igt at gem_exec_store@basic-bsd1
> +igt at gem_exec_store@basic-bsd2
> +igt at gem_exec_store@basic-default
> +igt at gem_exec_store@basic-render
> +igt at gem_exec_store@basic-vebox
> +igt at gem_exec_suspend@basic
> +igt at gem_exec_suspend@basic-s3
> +igt at gem_exec_suspend@basic-s4
> +igt at gem_flink_basic@bad-flink
> +igt at gem_flink_basic@bad-open
> +igt at gem_flink_basic@basic
> +igt at gem_flink_basic@double-flink
> +igt at gem_flink_basic@flink-lifetime
> +igt at gem_linear_blits@basic
> +igt at gem_mmap@basic
> +igt at gem_mmap@basic-small-bo
> +igt at gem_mmap_gtt@basic
> +igt at gem_mmap_gtt@basic-copy
> +igt at gem_mmap_gtt@basic-read
> +igt at gem_mmap_gtt@basic-read-no-prefault
> +igt at gem_mmap_gtt@basic-read-write
> +igt at gem_mmap_gtt@basic-read-write-distinct
> +igt at gem_mmap_gtt@basic-short
> +igt at gem_mmap_gtt@basic-small-bo
> +igt at gem_mmap_gtt@basic-small-bo-tiledx
> +igt at gem_mmap_gtt@basic-small-bo-tiledy
> +igt at gem_mmap_gtt@basic-small-copy
> +igt at gem_mmap_gtt@basic-small-copy-xy
> +igt at gem_mmap_gtt@basic-wc
> +igt at gem_mmap_gtt@basic-write
> +igt at gem_mmap_gtt@basic-write-cpu-read-gtt
> +igt at gem_mmap_gtt@basic-write-gtt
> +igt at gem_mmap_gtt@basic-write-gtt-no-prefault
> +igt at gem_mmap_gtt@basic-write-no-prefault
> +igt at gem_mmap_gtt@basic-write-read
> +igt at gem_mmap_gtt@basic-write-read-distinct
> +igt at gem_pread@basic
> +igt at gem_pwrite@basic
> +igt at gem_render_linear_blits@basic
> +igt at gem_render_tiled_blits@basic
> +igt at gem_ringfill@basic-default
> +igt at gem_ringfill@basic-default-forked
> +igt at gem_ringfill@basic-default-hang
> +igt at gem_ringfill@basic-default-interruptible
> +igt at gem_storedw_loop@basic-blt
> +igt at gem_storedw_loop@basic-bsd
> +igt at gem_storedw_loop@basic-bsd1
> +igt at gem_storedw_loop@basic-bsd2
> +igt at gem_storedw_loop@basic-default
> +igt at gem_storedw_loop@basic-render
> +igt at gem_storedw_loop@basic-vebox
> +igt at gem_sync@basic-all
> +igt at gem_sync@basic-each
> +igt at gem_sync@basic-many-each
> +igt at gem_sync@basic-store-all
> +igt at gem_sync@basic-store-each
> +igt at gem_tiled_blits@basic
> +igt at gem_tiled_fence_blits@basic
> +igt at gem_tiled_pread_basic
> +igt at gem_workarounds@basic-read
> +igt at gvt_basic@invalid-placeholder-test
> +igt at kms_addfb_basic@addfb25-bad-modifier
> +igt at kms_addfb_basic@addfb25-framebuffer-vs-set-tiling
> +igt at kms_addfb_basic@addfb25-modifier-no-flag
> +igt at kms_addfb_basic@addfb25-x-tiled
> +igt at kms_addfb_basic@addfb25-x-tiled-mismatch
> +igt at kms_addfb_basic@addfb25-yf-tiled
> +igt at kms_addfb_basic@addfb25-y-tiled
> +igt at kms_addfb_basic@addfb25-y-tiled-small
> +igt at kms_addfb_basic@bad-pitch-0
> +igt at kms_addfb_basic@bad-pitch-1024
> +igt at kms_addfb_basic@bad-pitch-128
> +igt at kms_addfb_basic@bad-pitch-256
> +igt at kms_addfb_basic@bad-pitch-32
> +igt at kms_addfb_basic@bad-pitch-63
> +igt at kms_addfb_basic@bad-pitch-65536
> +igt at kms_addfb_basic@bad-pitch-999
> +igt at kms_addfb_basic@basic
> +igt at kms_addfb_basic@basic-x-tiled
> +igt at kms_addfb_basic@basic-y-tiled
> +igt at kms_addfb_basic@bo-too-small
> +igt at kms_addfb_basic@bo-too-small-due-to-tiling
> +igt at kms_addfb_basic@clobberred-modifier
> +igt at kms_addfb_basic@framebuffer-vs-set-tiling
> +igt at kms_addfb_basic@invalid-get-prop
> +igt at kms_addfb_basic@invalid-get-prop-any
> +igt at kms_addfb_basic@invalid-set-prop
> +igt at kms_addfb_basic@invalid-set-prop-any
> +igt at kms_addfb_basic@no-handle
> +igt at kms_addfb_basic@size-max
> +igt at kms_addfb_basic@small-bo
> +igt at kms_addfb_basic@tile-pitch-mismatch
> +igt at kms_addfb_basic@too-high
> +igt at kms_addfb_basic@too-wide
> +igt at kms_addfb_basic@unused-handle
> +igt at kms_addfb_basic@unused-modifier
> +igt at kms_addfb_basic@unused-offsets
> +igt at kms_addfb_basic@unused-pitches
> +igt at kms_busy@basic-flip-default-a
> +igt at kms_busy@basic-flip-default-b
> +igt at kms_busy@basic-flip-default-c
> +igt at kms_cursor_legacy@basic-busy-flip-before-cursor-atomic
> +igt at kms_cursor_legacy@basic-busy-flip-before-cursor-legacy
> +igt at kms_cursor_legacy@basic-busy-flip-before-cursor-varying-size
> +igt at kms_cursor_legacy@basic-flip-after-cursor-atomic
> +igt at kms_cursor_legacy@basic-flip-after-cursor-legacy
> +igt at kms_cursor_legacy@basic-flip-after-cursor-varying-size
> +igt at kms_cursor_legacy@basic-flip-before-cursor-atomic
> +igt at kms_cursor_legacy@basic-flip-before-cursor-legacy
> +igt at kms_cursor_legacy@basic-flip-before-cursor-varying-size
> +igt at kms_flip@basic-flip-vs-dpms
> +igt at kms_flip@basic-flip-vs-modeset
> +igt at kms_flip@basic-flip-vs-wf_vblank
> +igt at kms_flip@basic-plain-flip
> +igt at kms_force_connector_basic@force-connector-state
> +igt at kms_force_connector_basic@force-edid
> +igt at kms_force_connector_basic@force-load-detect
> +igt at kms_force_connector_basic@prune-stale-modes
> +igt at kms_frontbuffer_tracking@basic
> +igt at kms_pipe_crc_basic@bad-nb-words-1
> +igt at kms_pipe_crc_basic@bad-nb-words-3
> +igt at kms_pipe_crc_basic@bad-pipe
> +igt at kms_pipe_crc_basic@bad-source
> +igt at kms_pipe_crc_basic@hang-read-crc-pipe-a
> +igt at kms_pipe_crc_basic@hang-read-crc-pipe-b
> +igt at kms_pipe_crc_basic@hang-read-crc-pipe-c
> +igt at kms_pipe_crc_basic@nonblocking-crc-pipe-a
> +igt at kms_pipe_crc_basic@nonblocking-crc-pipe-a-frame-sequence
> +igt at kms_pipe_crc_basic@nonblocking-crc-pipe-b
> +igt at kms_pipe_crc_basic@nonblocking-crc-pipe-b-frame-sequence
> +igt at kms_pipe_crc_basic@nonblocking-crc-pipe-c
> +igt at kms_pipe_crc_basic@nonblocking-crc-pipe-c-frame-sequence
> +igt at kms_pipe_crc_basic@read-crc-pipe-a
> +igt at kms_pipe_crc_basic@read-crc-pipe-a-frame-sequence
> +igt at kms_pipe_crc_basic@read-crc-pipe-b
> +igt at kms_pipe_crc_basic@read-crc-pipe-b-frame-sequence
> +igt at kms_pipe_crc_basic@read-crc-pipe-c
> +igt at kms_pipe_crc_basic@read-crc-pipe-c-frame-sequence
> +igt at kms_pipe_crc_basic@suspend-read-crc-pipe-a
> +igt at kms_pipe_crc_basic@suspend-read-crc-pipe-b
> +igt at kms_pipe_crc_basic@suspend-read-crc-pipe-c
> +igt at kms_psr_sink_crc@psr_basic
> +igt at kms_setmode@basic-clone-single-crtc
> +igt at kms_sink_crc_basic
> +igt at pm_backlight@basic-brightness
> +igt at pm_rpm@basic-pci-d3-state
> +igt at pm_rpm@basic-rte
> +igt at pm_rps@basic-api
> +igt at prime_busy@basic-after-default
> +igt at prime_busy@basic-before-default
> +igt at prime_busy@basic-wait-after-default
> +igt at prime_busy@basic-wait-before-default
> +igt at prime_self_import@basic-llseek-bad
> +igt at prime_self_import@basic-llseek-size
> +igt at prime_self_import@basic-with_fd_dup
> +igt at prime_self_import@basic-with_one_bo
> +igt at prime_self_import@basic-with_one_bo_two_files
> +igt at prime_self_import@basic-with_two_bos
> +igt at prime_vgem@basic-busy-default
> +igt at prime_vgem@basic-fence-flip
> +igt at prime_vgem@basic-fence-mmap
> +igt at prime_vgem@basic-fence-read
> +igt at prime_vgem@basic-fence-wait-default
> +igt at prime_vgem@basic-gtt
> +igt at prime_vgem@basic-read
> +igt at prime_vgem@basic-sync-default
> +igt at prime_vgem@basic-wait-default
> +igt at prime_vgem@basic-write
> +igt at vgem_basic@create
> +igt at vgem_basic@debugfs
> +igt at vgem_basic@dmabuf-export
> +igt at vgem_basic@dmabuf-fence
> +igt at vgem_basic@dmabuf-fence-before
> +igt at vgem_basic@dmabuf-mmap
> +igt at vgem_basic@mmap
> +igt at vgem_basic@second-client
> +igt at vgem_basic@sysfs
> +igt at vgem_basic@unload
> +igt at vgem_reload_basic