[igt-dev] [PATCH i-g-t v2 1/8] lib/tests: Fix test failures with meson 0.50.0

[Daniel Vetter]

On Tue, Apr 16, 2019 at 07:25:41PM -0400, Lyude wrote:
> From: Lyude Paul <lyude at redhat.com>
> 
> Since meson 0.50.0, unit tests which return the GNU standard return code
> 99 will fail, regardless of whether or not should_fail:true is passed to
> test(). Unfortunately, igt_alarm_handler() exits the application with
> return code 99. However, since returning something other then 99 when a
> test times out would also be a bug we can't really change the return
> code igt_alarm_handler() returns.
> 
> Instead, we can fix this by simply running tests which are expected to
> fail with return code 99 using a wrapper script that translates 99 to 1
> and any other non-zero error code into 99. This essentially makes it so
> that abnormal test failures are considered normal, and vice versa.
> 
> Signed-off-by: Lyude Paul <lyude at redhat.com>
> ---
>  lib/tests/meson.build           | 15 +++++++++++++++
>  lib/tests/retcode_99_wrapper.sh | 11 +++++++++++
>  2 files changed, 26 insertions(+)
>  create mode 100644 lib/tests/retcode_99_wrapper.sh
> 
> diff --git a/lib/tests/meson.build b/lib/tests/meson.build
> index 74efce39..108597cf 100644
> --- a/lib/tests/meson.build
> +++ b/lib/tests/meson.build
> @@ -18,6 +18,9 @@ lib_tests = [
>  lib_fail_tests = [
>  	'igt_no_subtest',
>  	'igt_simple_test_subtests',
> +]
> +
> +lib_retcode_99_tests = [
>  	'igt_timeout',

The usual way I've solved this is with fork, and an igt_internal_assert
that positively checks for the right waitpid status. The fail tests are
kinda a cheap hack, and if they don't work it's imo better to do proper
testing instead of wrapping more glue around it.

There's tons of examples if you look at git log lib/tests/
-Daniel

>  ]
>  
> @@ -32,3 +35,15 @@ foreach lib_test : lib_fail_tests
>  			dependencies : igt_deps)
>  	test('lib: ' + lib_test, exec, should_fail : true)
>  endforeach
> +
> +# Some tests are expected to fail with a retcode of 99. Since meson 0.50.0, unit
> +# tests that return this error code are marked as failures regardless of the
> +# value of should_fail. So, we run these tests in a wrapper to translate return
> +# codes of 99 into 1, and non-zero error codes into 99 (so normal failures are
> +# counted as abnormal failures)
> +wrapper_99 = find_program('retcode_99_wrapper.sh')
> +foreach lib_test : lib_retcode_99_tests
> +	exec = executable(lib_test, lib_test + '.c', install : false,
> +			dependencies : igt_deps)
> +	test('lib: ' + lib_test, wrapper_99, args : [exec], should_fail : true)
> +endforeach
> diff --git a/lib/tests/retcode_99_wrapper.sh b/lib/tests/retcode_99_wrapper.sh
> new file mode 100644
> index 00000000..cfd8825d
> --- /dev/null
> +++ b/lib/tests/retcode_99_wrapper.sh
> @@ -0,0 +1,11 @@
> +#!/bin/sh
> +
> +"$@"
> +last_ret="$?"
> +if [ $last_ret -eq 99 ]; then
> +	exit 1
> +elif [ $last_ret -ne 0 ]; then
> +	exit 99 # We expected to fail with 99, normal failures are abnormal
> +else
> +	exit $last_ret
> +fi
> -- 
> 2.20.1
> 
> _______________________________________________
> igt-dev mailing list
> igt-dev at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/igt-dev

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch