[Intel-gfx] [igt-dev] [PATH i-g-t] igt: Test tagging support

[Tvrtko Ursulin]

On 07/09/2018 12:43, Chris Wilson wrote:
> Quoting Tvrtko Ursulin (2018-09-07 12:14:20)
>> From: Tvrtko Ursulin <tvrtko.ursulin at intel.com>
>>
>> Proposal to add test tags as a replacement for separate test
>> list which can be difficult to maintain and get out of date.
>>
>> Putting this maintanenace inline with tests makes it easier
>> to remember to update the (now implicit) lists, and also enables
>> richer test selection possibilities for the test runner.
>>
>> Current method of implying tags from test/subtest names has a
>> couple of problems one of which is where some names can use
>> the same token for different meanings. (One example is the
>> "default" token.) It also creates a name clash between naming
>> and tagging.
>>
>> Test tags are strings of tokens separated by spaces, meaning of
>> which we decide by creating a convetion and helped by the
>> suitable helper macros.
>>
>> For example tags can be things like: gem, kms, basic, guc, flip,
>> semaphore, bz12345678, gt4e, reset, etc..
>>
>> At runtime this would look something like this (abbreviated for
>> readability):
>>
>>    @ tests/gem_sync --list-subtests-with-tags
>>    ...
>>    forked-each TAGS="gem "
>>    forked-store-each TAGS="gem "
>>    basic-all TAGS="gem basic "
>>    basic-store-all TAGS="gem basic "
>>
>>    @ tests/gem_concurrent_blit --list-subtests-with-tags
>>    ...
>>    16MiB-swap-gpuX-render-write-read-bcs-bomb TAGS="gem stress "
>>    16MiB-swap-gpuX-render-write-read-rcs-bomb TAGS="gem stress "
>>    16MiB-swap-gpuX-render-gpu-read-after-write-bomb TAGS="gem stress "
>>
>>    @ tests/kms_flip --list-subtests-with-tags | grep basic
>>    basic-plain-flip TAGS="kms basic "
>>    basic-flip-vs-dpms TAGS="kms basic "
>>
>> Test runner could then enable usages like:
>>
>>    ./run-tests --include gem --exclude stress
> 
> Minor comment, I like some basic boolean algebra here
> --include "gem AND smoketest NOT stress"
> :)

That's what my hypothetical examples showed just with a different syntax!

> I'd probably tag everything according to which ioctls they use. I feel my
> endgoal is still to list tests by their kernel functions (which can and
> should be automatically derived), and their hw function which is the
> open problem.

If we want to do that automatically then tagging in this flavour 
probably doesn't make sense and it should instead be an external database.

If we go on the ioctl granularity it can probably mostly have one 
version, and it should be static enough to be able to live in the tree, 
but if we go more granular, like something derived from kcov, then 
that's out of the window. Since it will be tied both to a platform and 
kernel version.

So I think tagging as proposed here is not appropriate for either, but 
only if we wanted to tag on coarser functional areas and such.

Regards,

Tvrtko