[PATCH i-g-t v2 0/2] lib/kunit: Execute test cases synchronously

[Cavitt, Jonathan]

All patches LGTM
Acked-by: Jonathan Cavitt <jonathan.cavitt at intel.com>
-Jonathan Cavitt

-----Original Message-----
From: Intel-gfx <intel-gfx-bounces at lists.freedesktop.org> On Behalf Of Janusz Krzysztofik
Sent: Tuesday, February 27, 2024 7:11 AM
To: igt-dev at lists.freedesktop.org
Cc: intel-gfx at lists.freedesktop.org; intel-xe at lists.freedesktop.org; Kamil Konieczny <kamil.konieczny at linux.intel.com>; Mauro Carvalho Chehab <mchehab at kernel.org>; De Marchi, Lucas <lucas.demarchi at intel.com>
Subject: [PATCH i-g-t v2 0/2] lib/kunit: Execute test cases synchronously
> 
> Up to now we were loading a KUnit test module in test execution mode only
> once per subtest, in background, and then, in parallel with execution of
> test cases while the module was loading, we were looking through dmesg for
> KTAP results from each expected test case.  As a consequence, our IGT
> messages were more or less delayed, never in full sync with kernel
> messages.  Moreover, parsing of KTAP results from already completed test
> cases could be abandoned on a failure from loading the test module or
> kernel taint caused by a subsequent test case.  Also, parsing of KTAP
> results from all subsequent test cases could be abandoned on a failure of
> the parser caused by any test case.  Other than that, if a user requested
> a single dynamic sub-subtest, all test cases were executed anyway while
> results from only one of them that corresponded to the selected dynamic
> sub-subtest were reported.  That way, kernel messages from unrelated test
> cases, not only the selected one, could contribute to dmesg-fail or dmesg-
> warn CI results from that sub-subtest.
> 
> Since recent KUnit implementation is capable of executing only those test
> cases that match a user filter, stop executing all of them asynchronously
> and parsing their KTAP results as they appear.  Instead, reload the test
> module once per each dynamic sub-subtest with a filter that selects a
> specific test case and wait for its completion.  If successful and no
> kernel taint has occurred then parse the whole KTAP report from a single
> test case it has produced and translate it to IGT result of that single
> corresponding sub-subtest.
> 
> v2: Refresh the series on top of changes to KUnit filters handling,
>   - move the code of a new helper from a previous patch 1 to a previous
>     patch 2 which now becomes patch 1,
>   - actually limit the scope of the helper to fetching a KTAP report from
>     a file descriptor, and let the callers decide on how other steps, like
>     setting up filters or loading a test module, and errors they return
>     are handled,
>   - update commit description with a more detailed justification of why we
>     need these changes,
>   - rebase the former patch 1 on top of the new patch 1, update its commit
>     message and description and provide it as patch 2.
> 
> Janusz Krzysztofik (2):
>   lib/kunit: Execute test cases synchronously
>   lib/kunit: Minimize code duplication
> 
>  lib/igt_kmod.c | 172 ++++++++++++++++---------------------------------
>  1 file changed, 54 insertions(+), 118 deletions(-)
> 
> -- 
> 2.43.0
> 
>