[Mesa-dev] [PATCH 0/4] glsl/tests: completely broken or strange design ?

[Dylan Baker]

Quoting Emil Velikov (2017-02-23 08:43:06)
> Hi all,
> 
> As some of you may know we have a bunch of interesting tests in glsl.
> 
> A bit earlier I've spotted that they are competely broken, as they would
> happily return "success" even when 0 tests were generated.
> 
> As an example I will take optimisation-test, which this series sort of
> fixes. Here is the design flow:
>  - optimisation-test (SH1) assumes that we're in the right location and 
> opens a python script (PY1).
>  - if found (only in corner cases) - the PY1 is executed and generates 
> a bunch of bash scripts (SH2) which are stored in the srcdir (fails on
> make distcheck).
>  - PY1 also hardcodes the location of the glsl_test binary, based on 
> some magically derived value (hint srcdir not used and wrong to use)
>  - the return status of PY1 is unchecked.
>  - thus as PY1 fails (and we often do) no tests are generated and SH1 
> reports "SUCCESS"
>  - alternatively, SH1 executes SH2 to produce the to-be-tested files
>  - then it (SH1) proceeds to execute PY2
>  - latter of which decodes and saves the [to a temporary location] the
> template and the file under test and calls "diff" on the resulting
> files.
> 
> With the following patches I've addressed all usecases, but the fact that
> srcdir is RO, thus 3/4 will cause "make distcheck" to fail (and rightfully so).
> 
> I believe all of us can see how fragile/suboptimal/you name it the
> design is and it need a complete rework.
> 
> Conteplating about giving a shot myself, but I suspect all of it will end up
> in the bin, now that we have Dylan back ;-)
> 
> So the questions are:
>  - Do we care about those tests ?
>  - Dylan any interest in giving it a stab ?

Not particularly, but I'd be happy to review whatever you come up with ;)

> 
> Thanks
> Emil
> 
> Cc: Dylan Baker <dylan at pnwbakers.com>
> Cc: Kenneth Graunke <kenneth at whitecape.org>
> Cc: Matt Turner <mattst88 at gmail.com>
> 
> Emil Velikov (4):
>   glsl/tests/optimisation-test: make sure that $PYTHON2 is set/available
>   glsl/tests/optimisation-test: ensure that compare_ir is available
>   glsl/tests/optimization-test: error out if we fail to generate any
>     tests
>   glsl/tests/optimization-test: correctly manage srcdir/pwd and co
> 
>  src/compiler/glsl/tests/optimization-test | 50 +++++++++++++++++++++++++++----
>  1 file changed, 44 insertions(+), 6 deletions(-)
> 
> -- 
> 2.11.0
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: signature
URL: <https://lists.freedesktop.org/archives/mesa-dev/attachments/20170224/ae00464b/attachment.sig>