[PATCH 0/3] Resubmit - Unit test framework for Wayland

[Eoff, Ullysses A]

>-----Original Message-----
>From: hoegsberg at gmail.com [mailto:hoegsberg at gmail.com] On Behalf Of
>Kristian Høgsberg
>Sent: Friday, March 02, 2012 5:17 PM
>
>Yeah, a few good points came up in this thread: process separation so
>if one test corrupts memory it doesn't affect others, being able to
>run all test even if some of them hangs/crashes, good summary of
>pass/fail.  And it turns out that looping through test cases and
>forking for each test case isn't a lot of code.  So I wrote that and
>converted Arties wl_array tests to this new test runner and added a
>test case for wl_map.  We can just use assert to log errors and
>segfaults are logged as errors automatically:  Output of make check
>looks like this (with an assert(0) in the array_copy test and a NULL
>pointer dererence added to the array_add test:
>
>running "array_copy"... array-test: array-test.c:121: array_copy: Assertion `0'\
> failed.
>signal 6
>running "array_add"... signal 11
>running "array_init"... exit status 0
>3 tests, 1 pass, 2 fail
>FAIL: array-test
>running "map_insert_new"... exit status 0
>1 tests, 1 pass, 0 fail
>PASS: map-test
>===========================================================
>================
>1 of 2 tests failed
>Please report to
>https://bugs.freedesktop.org/enter_bug.cgi?product=wayland
>===========================================================
>================
>
>As it is, the assert is printed in the middle of the "running ..." but
>I'll fix that so that it's printed on its own line, which means you
>can step through test failures like compile errors (at least in
>emacs).  And if you want to debug a specific case, run it as
>
>  $ gdb --args array-test array_add
>
>to run just that one case without forking.  There's only a couple of
>things that I think we might to add to this: 1) timeout for tests, but
>lets do that if we hit that case and 2) running multiple tests in
>parallel, but automake can help with that and as for 1), let's do that
>if we need it.
>
>The code for this is in tests/test-runner.c and it's all of 93 lines,
>including 22 lines of copyright header.  To write a test case do:
>
>TEST(name_of_test)
>{
>  assert(5 != 7);
>}
>
>link with test-runner.c, add the binary to TESTS and you're done.
>
>Kristian

Kristian,

I'm actually quite pleased that you found a solution to avoid having
to add/register your tests to a suite and/or test runner manually.  It
happens automatically... all you have to do is define your test; awesome!
That will definitely reduce the chance of human error  (i.e. writing a test
and then forgetting to add it to a suite/runner).  Let alone, it simplifies
test writing and maintenance, so all the developer has to do is focus on
the test case itself.  That was one of the top features I wanted in a test
framework.

I primarily have a C++ background and it was easy to find ways to do that
in that language.  When I realized that I couldn't figure out how to do it in C,
I was a little disappointed and just assumed it was a limitation of C and the
compiler.  Albeit, I didn't have a lot of time to investigate.

Kudos.  It's definitely a good start :)

I'm curious, however, as to why you've used relative file includes in the
test source files.  Why not just add the path to CFLAGS?  Also, any plans
to add the 'tests' directory to the top-level makefile SUBDIRS?

Artie