[igt-dev] [PATH i-g-t] igt: Test tagging support

[Chris Wilson]

Quoting Tvrtko Ursulin (2018-09-07 12:14:20)
> From: Tvrtko Ursulin <tvrtko.ursulin at intel.com>
> 
> Proposal to add test tags as a replacement for separate test
> list which can be difficult to maintain and get out of date.
> 
> Putting this maintanenace inline with tests makes it easier
> to remember to update the (now implicit) lists, and also enables
> richer test selection possibilities for the test runner.
> 
> Current method of implying tags from test/subtest names has a
> couple of problems one of which is where some names can use
> the same token for different meanings. (One example is the
> "default" token.) It also creates a name clash between naming
> and tagging.
> 
> Test tags are strings of tokens separated by spaces, meaning of
> which we decide by creating a convetion and helped by the
> suitable helper macros.
> 
> For example tags can be things like: gem, kms, basic, guc, flip,
> semaphore, bz12345678, gt4e, reset, etc..
> 
> At runtime this would look something like this (abbreviated for
> readability):
> 
>   @ tests/gem_sync --list-subtests-with-tags
>   ...
>   forked-each TAGS="gem "
>   forked-store-each TAGS="gem "
>   basic-all TAGS="gem basic "
>   basic-store-all TAGS="gem basic "
> 
>   @ tests/gem_concurrent_blit --list-subtests-with-tags
>   ...
>   16MiB-swap-gpuX-render-write-read-bcs-bomb TAGS="gem stress "
>   16MiB-swap-gpuX-render-write-read-rcs-bomb TAGS="gem stress "
>   16MiB-swap-gpuX-render-gpu-read-after-write-bomb TAGS="gem stress "
> 
>   @ tests/kms_flip --list-subtests-with-tags | grep basic
>   basic-plain-flip TAGS="kms basic "
>   basic-flip-vs-dpms TAGS="kms basic "
> 
> Test runner could then enable usages like:
> 
>   ./run-tests --include gem --exclude stress

Minor comment, I like some basic boolean algebra here
--include "gem AND smoketest NOT stress"
:)

I'd probably tag everything according to which ioctls they use. I feel my
endgoal is still to list tests by their kernel functions (which can and
should be automatically derived), and their hw function which is the
open problem.
-Chris