[Intel-gfx] [igt-dev] [PATH i-g-t] igt: Test tagging support

[Chris Wilson]

Quoting Tvrtko Ursulin (2018-09-12 09:48:00)
> 
> On 07/09/2018 12:43, Chris Wilson wrote:
> > Quoting Tvrtko Ursulin (2018-09-07 12:14:20)
> >> From: Tvrtko Ursulin <tvrtko.ursulin at intel.com>
> >>
> >> Proposal to add test tags as a replacement for separate test
> >> list which can be difficult to maintain and get out of date.
> >>
> >> Putting this maintanenace inline with tests makes it easier
> >> to remember to update the (now implicit) lists, and also enables
> >> richer test selection possibilities for the test runner.
> >>
> >> Current method of implying tags from test/subtest names has a
> >> couple of problems one of which is where some names can use
> >> the same token for different meanings. (One example is the
> >> "default" token.) It also creates a name clash between naming
> >> and tagging.
> >>
> >> Test tags are strings of tokens separated by spaces, meaning of
> >> which we decide by creating a convetion and helped by the
> >> suitable helper macros.
> >>
> >> For example tags can be things like: gem, kms, basic, guc, flip,
> >> semaphore, bz12345678, gt4e, reset, etc..
> >>
> >> At runtime this would look something like this (abbreviated for
> >> readability):
> >>
> >>    @ tests/gem_sync --list-subtests-with-tags
> >>    ...
> >>    forked-each TAGS="gem "
> >>    forked-store-each TAGS="gem "
> >>    basic-all TAGS="gem basic "
> >>    basic-store-all TAGS="gem basic "
> >>
> >>    @ tests/gem_concurrent_blit --list-subtests-with-tags
> >>    ...
> >>    16MiB-swap-gpuX-render-write-read-bcs-bomb TAGS="gem stress "
> >>    16MiB-swap-gpuX-render-write-read-rcs-bomb TAGS="gem stress "
> >>    16MiB-swap-gpuX-render-gpu-read-after-write-bomb TAGS="gem stress "
> >>
> >>    @ tests/kms_flip --list-subtests-with-tags | grep basic
> >>    basic-plain-flip TAGS="kms basic "
> >>    basic-flip-vs-dpms TAGS="kms basic "
> >>
> >> Test runner could then enable usages like:
> >>
> >>    ./run-tests --include gem --exclude stress
> > 
> > Minor comment, I like some basic boolean algebra here
> > --include "gem AND smoketest NOT stress"
> > :)
> 
> That's what my hypothetical examples showed just with a different syntax!
> 
> > I'd probably tag everything according to which ioctls they use. I feel my
> > endgoal is still to list tests by their kernel functions (which can and
> > should be automatically derived), and their hw function which is the
> > open problem.
> 
> If we want to do that automatically then tagging in this flavour 
> probably doesn't make sense and it should instead be an external database.
> 
> If we go on the ioctl granularity it can probably mostly have one 
> version, and it should be static enough to be able to live in the tree, 
> but if we go more granular, like something derived from kcov, then 
> that's out of the window. Since it will be tied both to a platform and 
> kernel version.
> 
> So I think tagging as proposed here is not appropriate for either, but 
> only if we wanted to tag on coarser functional areas and such.

Yeah, I think the same as well, that tags should be "smoke", "stress".
(But one man's stress is another's minimal workload :(

But both of those are in essence a duration parameter, and I'd prefer if
we expressed those as configurable parameters. At which point there is a
meta level of test scripts to tag ;)

I feel that "gem", "kms" is better expressed in lower level terms, so
what's left to tag? Clients? "display", "opencl", "media", "opengl" ?
Even using client specs for features (e.g. EGL_IMG_context_priority)?

Who and why would use those? From a selfish perspective, I want lcov
with tests sorted in order of increasing "stress" (start with the
precise X does Y test, finish with can it survive pathological client
behaviour for 48 hours).
-Chris