[Intel-gfx] [PATCH i-g-t 8/9] lib/kunit: Fetch a list of test cases in advance

[Janusz Krzysztofik]

Hi Kamil,

Thanks for review.

On Thursday, 5 October 2023 23:28:44 CEST Kamil Konieczny wrote:
> Hi Janusz,
> 
> On 2023-10-03 at 11:10:53 +0200, Janusz Krzysztofik wrote:
> > Recent improvements to the kernel kunit framework allow us to obtain a
> > list of test cases provided by a kunit test module without actually
> > running them.  Use that feature to get a list of expected test cases
> > before we enter a loop around igt_dynamic().  Once done, enter the
> > igt_dynamic() section for each consecutive test case immediately, even
> > before first line of a related KTAP report appears, then look for a result
> > from that test case.  That should make our IGT results output still better
> > synchronized with related kernel messages.
> > 
> > The list of test cases provided by a kunit test module can be obtained by
> > loading the kunit base module with specific options, then loading the test
> > module.  For that to be possible, take care of unloading the kunit base
> > module before each kunit subtest (I was wrong when in one of my previous
> > commit messages I suggested that on final unload of a kunit test module
> > the kunit base module is unloaded automatically as its dependency,
> > however, that didn't matter before, then no separate fix was required).
> > Since that module can then be left loaded with non-default options if an
> > error occurs, unload it explicitly before returning from igt_kunit().
> > 
> > There are two possible ways of getting a list of test cases: by loading
> > the base kunit module with action=list module option, or by filtering
> > out all test cases from being executed while asking for SKIP results from
> > those filtered out.  Since the latter provides regular KTAP report that we
> > can alredy parse perfectly, use it instead of trying to identify an
> ----- ^^^^^^
> s/alredy/already/

Thanks.

> 
> > unstructured list of test cases of unknown length submitted by the former.
> > 
> > If an IGT test that calls igt_kunit() provides a subtest name then use
> > that name to filter out potential test cases that don't belong to the
> > named test suite from the list.
> > 
> > To avoid loading any modules if no subtest is going to be executed (e.g.,
> > if a nonexistent subtest has been requested), load the kunit modules in
> > list mode from inside the igt_subtest_with_dynamic() section.  In order to
> > be free to skip the whole subtest on unmet requirements that need to be
> > verified after that list has been already populated, clean it up from a
> > follow up igt_fixture section.
> > 
> > Since we start reading the list of test cases from /dev/kmsg only after
> > the kunit test module is loaded successfully in list only mode, don't
> > synchronize reads with potential modprobe breakage in that case, unlike we
> > still do later when parsing KTAP results in parallel to loading the test
> > module in normal (execute) mode.
> > 
> > Since we neither fetch KTAP results before entering igt_dynamic section
> > nor even return an error from KTAP result fetch attempts immediately on
> > modprobe error or kernel taint, break the loop of dynamic sub-subtests
> > explicitly as soon as one of those conditions is detected.
> 
> Could you split this patch into two, one with reading names and rest
> with modprobe error or kernel taint detection?

Yes, sounds like a good idea.

> 
> > Also, don't
> > force IGT SKIP result from the subtest if KTAP parsing hasn't completed.
> > That's perfectly legitimate since we no longer iterate over KTAP results,
> > only over a list of test cases obtained in advance, then we stop parsing
> > KTAP report as soon as we get a result from the last test case from the
> > list.
> > 
> > Signed-off-by: Janusz Krzysztofik <janusz.krzysztofik at linux.intel.com>
> > ---
> >  lib/igt_kmod.c | 217 +++++++++++++++++++++++++++++++++++--------------
> >  1 file changed, 157 insertions(+), 60 deletions(-)
> > 
> > diff --git a/lib/igt_kmod.c b/lib/igt_kmod.c
> > index 387efbb59f..4fba77ead4 100644
> > --- a/lib/igt_kmod.c
> > +++ b/lib/igt_kmod.c
...  
> > @@ -905,6 +905,63 @@ __igt_kunit(struct igt_ktest *tst, const char *name, const char *opts)
> >  
> >  	igt_skip_on(lseek(tst->kmsg, 0, SEEK_END) < 0);
> >  
> > +	/*
> > +	 * To get a list of test cases provided by a kunit test module, ask the
> > +	 * generic kunit module to respond with SKIP result for each test found.
> > +	 * We could also use action=list kunit parameter to get the listing,
> > +	 * however, parsing a KTAP report -- something that we alredy can do
> --------------------------------------------------------------^
> Same here.

One more thank you.
Janusz