[Intel-gfx] [PATCH i-g-t] lib/igt_core.c: Expand --run-subtest functionality.

[Morton, Derek J]

>
>
>-----Original Message-----
>From: Daniel Vetter [mailto:daniel.vetter at ffwll.ch] On Behalf Of Daniel Vetter
>Sent: Wednesday, January 27, 2016 3:43 PM
>To: Morton, Derek J
>Cc: Daniel Vetter; Ville Syrjälä; intel-gfx at lists.freedesktop.org
>Subject: Re: [Intel-gfx] [PATCH i-g-t] lib/igt_core.c: Expand --run-subtest functionality.
>
>On Wed, Jan 27, 2016 at 02:01:36PM +0000, Morton, Derek J wrote:
>> >
>> >-----Original Message-----
>> >From: Daniel Vetter [mailto:daniel.vetter at ffwll.ch] On Behalf Of 
>> >Daniel Vetter
>> >Sent: Wednesday, January 27, 2016 1:39 PM
>> >To: Ville Syrjälä
>> >Cc: Morton, Derek J; intel-gfx at lists.freedesktop.org
>> >Subject: Re: [Intel-gfx] [PATCH i-g-t] lib/igt_core.c: Expand --run-subtest functionality.
>> >
>> >On Wed, Jan 27, 2016 at 02:32:47PM +0200, Ville Syrjälä wrote:
>> >> On Wed, Jan 27, 2016 at 10:05:56AM +0000, Derek Morton wrote:
>> >> > Added support for specifying arbitary lists of subtests to run or 
>> >> > to exclude from being run by using : or ^ as a seperator.
>> >> > 
>> >> > :subtest1:subtest2: Will run subtest1 and subtest2 
>> >> > ^subtest1^subtest2^ will run all subtests except subtest1 and
>> >> > subtest2
>> >> 
>> >> Hmm. Getting a bit complicated perhaps. Would it be simpler to just 
>> >> allow specifying the --r option multiple times? So we'd start with 
>> >> the full list of subtests, and each --r option would filter the 
>> >> list in some way?
>> >
>> >Also why not use piglit ... or what is this for?
>> 
>> We don't use piglet on android. Piglet does not know about adb. Piglet 
>> expects to be running on the system under test not on a separate host.
>
>This can be fixed, and iirc there's even been patches floating around to run piglits remotely via adb.
>
>> The main aim of this is because on android we are not testing a driver 
>> which is drm-nightly. The kernel / display driver used on android will 
>> not have all the features that are in the latest linux kernel. We keep 
>> hitting problems where new subtests get added to IGT to test features 
>> that are not yet in the android kernel. We run and report tests at a 
>> binary level as that is what the project managers expect. We wish to 
>> be able to run the latest versions of IGT to pick up bug fixes and 
>> useful test changes, but want a way of being able to exclude subtests 
>> that are not currently appropriate on android without having to 
>> exclude complete test binaries. The specific subtests which need to be 
>> excluded will differ depending on the HW (CHV vs BXT for example) and 
>> specific driver version in the build under test so we need a simple 
>> mechanism to specify the subtests to run or exclude (depending on 
>> which is more appropriate) for each test.
>
>So a bunch of things:
>- Reporting at the per-binary level. Still doesn't make sense, and really
>  imo not a technical issue. Worst case write shell scripts (or
>  autogenerate those) with the testcase groups.

Not a technical issue, but easier to add additional functionality to igt than write scripts to execute the binary multiple times. Memory leaks between subtests will be hidden if the binary is executed separately for each subtest. Anything in fixtures are executed once per binary and not once per subtest so it is possible at least to get some strange behaviour difference. In either case if there is a difference between running subtests separately or together it would be a bug in the test code.

There would be some overhead in scripting by the need to call with --list-subtests then remove the undesired subtests before running the rest of the subtests. Quite a lot of work compared to the size of this patch putting the functionality directly in IGT.

>
>- igts falling over when the kernel doesn't support a feature. This
>  shouldn't ever happen, igt testcases are suppose to skip when the
>  requirements aren't met. Please report any such cases so that we can fix
>  them up in upstream igt.

I do not think everything is fixable upstream.

We have had cases where there is an ioctl missing on android (or worse does something different, though that gets fixed eventually), or where an ioctl has been extended. If the ioctl fails the test fails.

Generally if we find an issue we think is fixable in upstream IGT we will do so.

//Derek


>
>- Android folks breaking the libdrm abi isn't in your list, but comes up
>  fairly often, too.
>
>Cheers, Daniel
>--
>Daniel Vetter
>Software Engineer, Intel Corporation
>http://blog.ffwll.ch
>