[PATCH i-g-t v3 0/5] lib/kunit: Execute test cases synchronously

[Cavitt, Jonathan]

-----Original Message-----
From: Janusz Krzysztofik <janusz.krzysztofik at linux.intel.com> 
Sent: Monday, March 18, 2024 3:13 AM
To: igt-dev at lists.freedesktop.org
Cc: intel-gfx at lists.freedesktop.org; intel-xe at lists.freedesktop.org; Kamil Konieczny <kamil.konieczny at linux.intel.com>; Mauro Carvalho Chehab <mchehab at kernel.org>; Cavitt, Jonathan <jonathan.cavitt at intel.com>; De Marchi, Lucas <lucas.demarchi at intel.com>
Subject: [PATCH i-g-t v3 0/5] lib/kunit: Execute test cases synchronously
> 
> Up to now we were loading a KUnit test module in test execution mode only
> once per subtest, in background, and then, in parallel with execution of
> test cases while the module was loading, we were looking through dmesg for
> KTAP results from each expected test case.  As a consequence, our IGT
> messages were more or less delayed, never in full sync with kernel
> messages.  Moreover, parsing of KTAP results from already completed test
> cases could be abandoned on a failure from loading the test module or
> kernel taint caused by a subsequent test case.  Also, parsing of KTAP
> results from all subsequent test cases could be abandoned on a failure of
> the parser caused by any test case.  Other than that, if a user requested
> a single dynamic sub-subtest, all test cases were executed anyway while
> results from only one of them that corresponded to the selected dynamic
> sub-subtest were reported.  That way, kernel messages from unrelated test
> cases, not only the selected one, could contribute to dmesg-fail or dmesg-
> warn CI results from that sub-subtest.
> 
> Since recent KUnit implementation is capable of executing only those test
> cases that match a user filter, stop executing all of them asynchronously
> and parsing their KTAP results as they appear.  Instead, reload the test
> module once per each dynamic sub-subtest with a filter that selects a
> specific test case and wait for its completion.  If successful and no
> kernel taint has occurred then parse the whole KTAP report from a single
> test case it has produced and translate it to IGT result of that single
> corresponding sub-subtest.
> 
> v3: Insert new patches 1-3 that fix an infinite loop when we try to get a
>     list of test cases from an unexpectedly missing KTAP report.
> v2: Refresh the series on top of changes to KUnit filters handling,
>   - move the code of a new helper from a previous patch 1 to a previous
>     patch 2 which now becomes patch 1,
>   - actually limit the scope of the helper to fetching a KTAP report from
>     a file descriptor, and let the callers decide on how other steps, like
>     setting up filters or loading a test module, and errors they return
>     are handled,
>   - update commit description with a more detailed justification of why we
>     need these changes,
>   - rebase the former patch 1 on top of the new patch 1, update its commit
>     message and description and provide it as patch 2.
> 
> Janusz Krzysztofik (5):
>   lib/kunit: Store igt_ktap_results pointer in a central location
>   lib/kunit: Let igt_ktap_free() take care of pointer reset
>   lib/kunit: Time out promptly on missing KTAP report
>   lib/kunit: Execute test cases synchronously
>   lib/kunit: Minimize code duplication
> 


Acked-by: Jonathan Cavitt <jonathan.cavitt at intel.com>
Ack applies to all patches in series.
It seems that Kamil is giving a proper review to all the individual patches,
so I'd wait until that's done before moving forward.
-Jonathan Cavitt


>  lib/igt_kmod.c              | 193 ++++++++++++++----------------------
>  lib/igt_ktap.c              |   5 +-
>  lib/igt_ktap.h              |   2 +-
>  lib/tests/igt_ktap_parser.c |  24 ++---
>  4 files changed, 93 insertions(+), 131 deletions(-)
> 
> -- 
> 2.43.0
> 
>