[Piglit] [PATCH] RFC: README.md: update test sets documentation

[Dylan Baker]

Quoting Alejandro Piñeiro (2019-06-03 05:41:53)
> Since commit "tests: use meta profiles"
> (400dcd5cf18ac51ab8e9681d8ecab4132986efbd) some test sets are not
> represented anymore as python scripts. So first, it would be good to
> mention it, and second it would be good to mention that it would be
> better to run the test sets using the set name, not directly the
> python script.
> 
> RFC because I have been struggling with the wording.
> ---
> 
> Basically I was not understanding why ./piglit run tests/all.py was
> not working, and the README.md didn't help me. So somehow it needs to
> be tweaked, but not sure how. Current patch tries to be as small as
> possible, but perhaps it is too small.
> 
> 
>  README.md | 24 +++++++++++++-----------
>  1 file changed, 13 insertions(+), 11 deletions(-)
> 
> diff --git a/README.md b/README.md
> index 7366893f8..b3267cd58 100644
> --- a/README.md
> +++ b/README.md
> @@ -313,26 +313,28 @@ available, then this completion file will be installed system-wide.
>  
>  ## 4. Available test sets
>  
> -Test sets are specified as Python scripts in the tests directory.
> -The following test sets are currently available:
> +Test sets are specified as Python scripts in the tests directory or as
> +meta profiles (for that reason it is better to run the test suite
   ^^^^
   I'd say xml here instead of meta, meta profiles are just xml that describe
   additional xml to load
> +name, not the python script).  The following test sets are currently
> +available:
>  
>  
>  ### 4.1 OpenGL Tests
>  
> -  - **sanity.py** This suite contains minimal OpenGL sanity tests. These tests
> +  - **sanity** This suite contains minimal OpenGL sanity tests. These tests
>      must pass, otherwise the other tests will not generate reliable results.
> -  - **all.py** This suite contains all OpenGL tests.
> -  - **quick.py** Run all tests, but cut down significantly on their runtime
> +  - **all** This suite contains all OpenGL tests.
> +  - **quick** Run all tests, but cut down significantly on their runtime
>      (and thus on the number of problems they can find).
> -  - **gpu.py** A further reduced set of tests from quick.py, this runs tests
> +  - **gpu** A further reduced set of tests from quick.py, this runs tests
>      only for hardware functionality and not tests for the software stack.
> -  - **llvmpipe.py** A reduced set of tests from gpu.py removing tests that are
> +  - **llvmpipe** A reduced set of tests from gpu.py removing tests that are
>      problematic using llvmpipe
> -  - **cpu.py** This profile runs tests that don't touch the gpu, in other words
> +  - **all** This suite contains all OpenGL tests.
> +  - **quick** Run all tests, but cut down significantly on their runtime
> +  - **gpu** A further reduced set of tests from quick.py, this runs tests
> +  - **llvmpipe** A reduced set of tests from gpu.py removing tests that are
> +  - **cpu** This profile runs tests that don't touch the gpu, in other words

cpu is just an alias of glslparser at this point.

I think it would be good to describe these as they are implemented, not that
glslpasers is a subset of all, but quick is quick_gl + quick_shader + glslparser

>      all of the tests in quick.py that are not run by gpu.py
> -  - **glslparser.py** A subset of all.py which runs only glslparser tests
> -  - **shader.py** A subset of all.py which runs only shader tests
> -  - **no_error.py** A modified version of the test list run as khr_no_error
> +  - **glslparser** A subset of all.py which runs only glslparser tests
> +  - **shader** A subset of all.py which runs only shader tests
> +  - **no_error** A modified version of the test list run as khr_no_error
>      variants
>  
>  
> -- 
> 2.19.1
> 

Gah, I never updated that did I?

I'll try to outline the logic as best I can:
  - *.xml.gz a single test profile
  - *.meta.xml a set of profiles glued together to maintain interfaces
  - *.py external profiles and profiles before being.xml'd

Historically the meta profiles were a single profile, and then subsets were
generated by filtering them down (quick was a subset of all, gpu was a subset of
quick, shader was a subset of quick), but I flipped that relationship both to
simplify the implementation and to make the build quicker (quick without shader
tests has no dependencies, but the shader portion depends on shader generators,
for example).

Dylan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: signature
URL: <https://lists.freedesktop.org/archives/piglit/attachments/20190604/b9fe4a72/attachment.sig>