[PATCH 0/3] Resubmit - Unit test framework for Wayland

[Kristian Høgsberg]

On Fri, Mar 2, 2012 at 5:28 PM, Daniel Stone <daniel at fooishbar.org> wrote:
> Hi,
>
> On 2 March 2012 21:53, Andreas Ericsson <ae at op5.se> wrote:
>> I've written a few in my days. Normally, I keep them ridiculously
>> simple, so the testing code looks something like this (sorry for the
>> sucky indentation; coding in a mua is always crap):
>>
>> some_test_func(args)
>> {
>>        test_suite t;
>>        int x, y;
>>
>>        x = 5;
>>        y = x;
>>        test(&t, x == y, "x(%d) and y(%d) should be equal", x, y);
>>        stest(&t, x == y); /* would print "fail: x == y evaluated as FALSE" */
>>        end_tests(&t); /* would print "OK: %d/%d tests passed */
>> }
>
> Maybe it's just a poor example, but something like this:
> some_test_func(args)
> {
>    int x = 5;
>    int y = 5;
>
>    printf("checking if C compiler isn't thoroughly broken ... ");
>    assert(x == y);
>    printf("success\n");
> }
>
> would seem to work just as well.
>
> I can almost see the argument for something slightly more
> comprehensive though, since rendercheck-style results of being able to
> compare various runs in their entirety rather than bailing out on the
> first failure, are more useful.  But still ...

Yeah, a few good points came up in this thread: process separation so
if one test corrupts memory it doesn't affect others, being able to
run all test even if some of them hangs/crashes, good summary of
pass/fail.  And it turns out that looping through test cases and
forking for each test case isn't a lot of code.  So I wrote that and
converted Arties wl_array tests to this new test runner and added a
test case for wl_map.  We can just use assert to log errors and
segfaults are logged as errors automatically:  Output of make check
looks like this (with an assert(0) in the array_copy test and a NULL
pointer dererence added to the array_add test:

running "array_copy"... array-test: array-test.c:121: array_copy: Assertion `0'\
 failed.
signal 6
running "array_add"... signal 11
running "array_init"... exit status 0
3 tests, 1 pass, 2 fail
FAIL: array-test
running "map_insert_new"... exit status 0
1 tests, 1 pass, 0 fail
PASS: map-test
===========================================================================
1 of 2 tests failed
Please report to https://bugs.freedesktop.org/enter_bug.cgi?product=wayland
===========================================================================

As it is, the assert is printed in the middle of the "running ..." but
I'll fix that so that it's printed on its own line, which means you
can step through test failures like compile errors (at least in
emacs).  And if you want to debug a specific case, run it as

  $ gdb --args array-test array_add

to run just that one case without forking.  There's only a couple of
things that I think we might to add to this: 1) timeout for tests, but
lets do that if we hit that case and 2) running multiple tests in
parallel, but automake can help with that and as for 1), let's do that
if we need it.

The code for this is in tests/test-runner.c and it's all of 93 lines,
including 22 lines of copyright header.  To write a test case do:

TEST(name_of_test)
{
  assert(5 != 7);
}

link with test-runner.c, add the binary to TESTS and you're done.

Kristian