[Bug 52049] Update telepathy-gabble-tests to show pass/fail tests status

[bugzilla-daemon at freedesktop.org]

https://bugs.freedesktop.org/show_bug.cgi?id=52049

--- Comment #3 from Luis Araujo <luis.araujo at collabora.co.uk> 2012-07-20 15:09:40 PDT ---
(In reply to comment #1)
> If I was running the installed tests, I wouldn't want to have to grep the
> results to find out which ones passed/failed. The script should exit nonzero if
> a test fails.
>

I guess this could also depend on the automated testing framework, talking more
specifically about LAVA, the way it works is parsing the tests output, checking
for 'pass' , 'fail' statuses and publishing meaningful information using this
data to a dashboard. So I wrote this patch thinking in that case mainly. 

> There are two ways this could go: either exit nonzero as soon as a test fails,
> or carry on testing, and exit nonzero if at least one test failed.
> 

A proper exit code is always good, though from a LAVA point of view the exit
code is not really important to process the tests data, so itis good and fine
if you add a final non-zero exit code if any test fail, though the test should
continue until the very end, a 'fail' status for a test is actually a valid
result for LAVA , so even if all tests fail, they should be run and published
to get this data.

> Also, exiting with status 77 is special in Automake tests: it indicates that
> the test was skipped for some reason (neither success nor failure).
> 

This is good, if you add a SKIP result to the script, we could parse this from
LAVA too; basically, we can parse and map ANY_STATUS as long as it properly
shows in the output of the test.

> I'd prefer something like this:
> 
> any_failed=0
> for i in $list; do
>     ... run the test ...
>     e=$?
>     case "$e" in
>         (0)
>             echo "Result  $i: pass"
>             ;;
>         (77)
>             echo "Result  $i: skip"
>             ;;
>         (*)
>             any_failed=1
>             echo "Result  $i: fail ($e)"
>             ;;
>     esac
> done
> 
> exit $any_failed
> 

This is very good++

> (Or if you want to exit as soon as a test fails, $any_failed is not needed, and
> you can exit from the failure "case".)

Let's run all the tests, after all, a fail value is a valid test data, so above
script would make it very good too.

-- 
Configure bugmail: https://bugs.freedesktop.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA Contact for the bug.
You are the assignee for the bug.