[PATCH i-g-t v7 11/16] scripts/test_list: Relax treatment of non-compiled tests

[Zbigniew Kempczyński]

On Wed, Sep 18, 2024 at 01:30:12PM +0200, Christoph Manszewski wrote:
> From: Kamil Konieczny <kamil.konieczny at linux.intel.com>
> 
> Some tests could be added and compiled only when non-default
> meson option is given. It will result in no testlist generated
> for such tests but they could appear in our documentation.
> Currently we report an error in such case.
> 
> Create a way to detect and report that and treat this as feature
> and do not report it as an error.  This should allow to relax
> processing of chamelium and other new tests.
> 
> Also while at this, print error information before actually
> exiting with an error code, as previous prints were only
> warnings.
> 
> Cc: Christoph Manszewski <christoph.manszewski at intel.com>
> Cc: Zbigniew Kempczyński <zbigniew.kempczynski at intel.com>
> Cc: Jari Tahvanainen <jari.tahvanainen at intel.com>
> Cc: Katarzyna Piecielska <katarzyna.piecielska at intel.com>
> Cc: Mauro Carvalho Chehab <mchehab at kernel.org>
> Signed-off-by: Kamil Konieczny <kamil.konieczny at linux.intel.com>
> Signed-off-by: Christoph Manszewski <christoph.manszewski at intel.com>

Works for me:

Reviewed-by: Zbigniew Kempczyński <zbigniew.kempczynski at intel.com>

--
Zbigniew

> ---
>  scripts/test_list.py | 32 ++++++++++++++++++++++++--------
>  1 file changed, 24 insertions(+), 8 deletions(-)
> 
> diff --git a/scripts/test_list.py b/scripts/test_list.py
> index 69c830ca1..d050687fe 100644
> --- a/scripts/test_list.py
> +++ b/scripts/test_list.py
> @@ -1141,6 +1141,19 @@ class TestList:
>  
>          return sorted(tests)
>  
> +    def get_not_compiled(self):
> +
> +        """ Return a list of tests which were not compiled """
> +        no_binaries = []
> +        for name in self.filenames:
> +            test_basename = re.sub(r"\.c$", "", name.split('/')[-1])
> +            fname = os.path.join(self.igt_build_path, "tests", test_basename)
> +
> +            if not os.path.isfile(fname):
> +                no_binaries.append(test_basename)
> +
> +        return sorted(no_binaries)
> +
>      #
>      # Validation methods
>      #
> @@ -1177,21 +1190,24 @@ class TestList:
>          # Get a list of tests from
>          run_subtests = set(self.get_testlist())
>  
> +        not_compiled = set(self.get_not_compiled())
> +        for test_basename in not_compiled:
> +            print(f"INFO: Found documentation for '{test_basename}' but no binary")
> +
>          # Compare sets
>  
>          run_missing = list(sorted(run_subtests - doc_subtests))
>          doc_uneeded = list(sorted(doc_subtests - run_subtests))
> +        doc_uneeded_build = [t for t in doc_uneeded if t.split('@')[1] not in not_compiled]
>  
> -        if doc_uneeded:
> -            for test_name in doc_uneeded:
> -                print(f"Warning: Documented {test_name} doesn't exist on source files")
> -
> -        if run_missing:
> +        if doc_uneeded_build or run_missing:
>              for test_name in run_missing:
> -                print(f'Warning: Missing documentation for {test_name}')
> -            print('Please refer: docs/test_documentation.md for more details')
> +                print(f'ERROR: Missing documentation for {test_name}')
>  
> -        if doc_uneeded or run_missing:
> +            for test_name in doc_uneeded_build:
> +                print(f'ERROR: Unneeded documentation for {test_name}')
> +
> +            print('Please refer: docs/test_documentation.md for more details')
>              sys.exit(1)
>  
>      #
> -- 
> 2.34.1
>