[Intel-gfx] [PATCH i-g-t] lib/igt_core.c: Expand --run-subtest functionality.

[Daniel Vetter]

On Wed, Jan 27, 2016 at 02:01:36PM +0000, Morton, Derek J wrote:
> >
> >-----Original Message-----
> >From: Daniel Vetter [mailto:daniel.vetter at ffwll.ch] On Behalf Of Daniel Vetter
> >Sent: Wednesday, January 27, 2016 1:39 PM
> >To: Ville Syrjälä
> >Cc: Morton, Derek J; intel-gfx at lists.freedesktop.org
> >Subject: Re: [Intel-gfx] [PATCH i-g-t] lib/igt_core.c: Expand --run-subtest functionality.
> >
> >On Wed, Jan 27, 2016 at 02:32:47PM +0200, Ville Syrjälä wrote:
> >> On Wed, Jan 27, 2016 at 10:05:56AM +0000, Derek Morton wrote:
> >> > Added support for specifying arbitary lists of subtests to run or to 
> >> > exclude from being run by using : or ^ as a seperator.
> >> > 
> >> > :subtest1:subtest2: Will run subtest1 and subtest2 
> >> > ^subtest1^subtest2^ will run all subtests except subtest1 and 
> >> > subtest2
> >> 
> >> Hmm. Getting a bit complicated perhaps. Would it be simpler to just 
> >> allow specifying the --r option multiple times? So we'd start with the 
> >> full list of subtests, and each --r option would filter the list in 
> >> some way?
> >
> >Also why not use piglit ... or what is this for?
> 
> We don't use piglet on android. Piglet does not know about adb. Piglet
> expects to be running on the system under test not on a separate host.

This can be fixed, and iirc there's even been patches floating around to
run piglits remotely via adb.

> The main aim of this is because on android we are not testing a driver
> which is drm-nightly. The kernel / display driver used on android will
> not have all the features that are in the latest linux kernel. We keep
> hitting problems where new subtests get added to IGT to test features
> that are not yet in the android kernel. We run and report tests at a
> binary level as that is what the project managers expect. We wish to be
> able to run the latest versions of IGT to pick up bug fixes and useful
> test changes, but want a way of being able to exclude subtests that are
> not currently appropriate on android without having to exclude complete
> test binaries. The specific subtests which need to be excluded will
> differ depending on the HW (CHV vs BXT for example) and specific driver
> version in the build under test so we need a simple mechanism to specify
> the subtests to run or exclude (depending on which is more appropriate)
> for each test.

So a bunch of things:
- Reporting at the per-binary level. Still doesn't make sense, and really
  imo not a technical issue. Worst case write shell scripts (or
  autogenerate those) with the testcase groups.

- igts falling over when the kernel doesn't support a feature. This
  shouldn't ever happen, igt testcases are suppose to skip when the
  requirements aren't met. Please report any such cases so that we can fix
  them up in upstream igt.

- Android folks breaking the libdrm abi isn't in your list, but comes up
  fairly often, too.

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch