[PATCH weston] Added simple unit/integration test framework and corresponding test program.

[Pekka Paalanen]

On Wed, 29 Apr 2015 15:31:45 -0700
"Jon A. Cruz" <jonc at osg.samsung.com> wrote:

> Added a simple C-based test framework and an example program
> that uses it to run through some simple wayland client checks.
> 
> Signed-off-by: Jon A. Cruz <jonc at osg.samsung.com>
> ---
>  .gitignore                            |   3 +
>  Makefile.am                           |  34 +++
>  tools/Doxyfile                        | 318 +++++++++++++++++++++
>  tools/devdocs.doxyfile                | 314 +++++++++++++++++++++
>  tools/tools.dox                       |  29 ++
>  tools/waycheck/rough_draw.c           | 176 ++++++++++++
>  tools/waycheck/rough_draw.h           |  42 +++
>  tools/waycheck/waycheck.c             | 505 ++++++++++++++++++++++++++++++++++
>  tools/waycheck/waycheck.dox           |  29 ++
>  tools/waycheck/wtst_fixtures.c        | 289 +++++++++++++++++++
>  tools/waycheck/wtst_fixtures.h        | 144 ++++++++++
>  tools/zunitc/doc/zunitc.dox           |  67 +++++
>  tools/zunitc/inc/zunitc/zunitc.h      | 300 ++++++++++++++++++++
>  tools/zunitc/inc/zunitc/zunitc_impl.h |  76 +++++
>  tools/zunitc/src/zunitc_impl.c        | 496 +++++++++++++++++++++++++++++++++
>  tools/zunitc/test/zunitc_test.c       | 106 +++++++
>  16 files changed, 2928 insertions(+)
>  create mode 100644 tools/Doxyfile
>  create mode 100644 tools/devdocs.doxyfile
>  create mode 100644 tools/tools.dox
>  create mode 100644 tools/waycheck/rough_draw.c
>  create mode 100644 tools/waycheck/rough_draw.h
>  create mode 100644 tools/waycheck/waycheck.c
>  create mode 100644 tools/waycheck/waycheck.dox
>  create mode 100644 tools/waycheck/wtst_fixtures.c
>  create mode 100644 tools/waycheck/wtst_fixtures.h
>  create mode 100644 tools/zunitc/doc/zunitc.dox
>  create mode 100644 tools/zunitc/inc/zunitc/zunitc.h
>  create mode 100644 tools/zunitc/inc/zunitc/zunitc_impl.h
>  create mode 100644 tools/zunitc/src/zunitc_impl.c
>  create mode 100644 tools/zunitc/test/zunitc_test.c
> 

Hi Jon,

I said I didn't have time to look at this, but I decided to take a
peek as "bed time reading", just to see what you were up to.

It seems that all this patch does is duplicate perhaps half
of what our ad hoc testing framework already does. It is not
without merits though, more on that below.

> diff --git a/tools/zunitc/doc/zunitc.dox b/tools/zunitc/doc/zunitc.dox
> new file mode 100644
> index 0000000..136d926
> --- /dev/null
> +++ b/tools/zunitc/doc/zunitc.dox
> @@ -0,0 +1,67 @@
> +/*
> + * Copyright © 2015 Samsung Electronics
> + *
> + * Permission to use, copy, modify, distribute, and sell this software and
> + * its documentation for any purpose is hereby granted without fee, provided
> + * that the above copyright notice appear in all copies and that both that
> + * copyright notice and this permission notice appear in supporting
> + * documentation, and that the name of the copyright holders not be used in
> + * advertising or publicity pertaining to distribution of the software
> + * without specific, written prior permission.  The copyright holders make
> + * no representations about the suitability of this software for any
> + * purpose.  It is provided "as is" without express or implied warranty.
> + *
> + * THE COPYRIGHT HOLDERS DISCLAIM ALL WARRANTIES WITH REGARD TO THIS
> + * SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
> + * FITNESS, IN NO EVENT SHALL THE COPYRIGHT HOLDERS BE LIABLE FOR ANY
> + * SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
> + * RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
> + * CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
> + * CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
> + */
> +
> +/**
> + at page zunitc zunitc
> +
> +A simple test framework in plain C suitable for basic unit and integration testing.
> +
> +The main rationale for creating this framework was to have a simple to use testing
> +framework with tests implemented in C using common patterns and under a
> +compatible license. The structure of the test code and macro use is intended to
> +follow common patterns established by frameworks such as Boost Test and Google Test.

Doesn't seem too different from what we had already...

> +To get started, one or more tests should be defined via ZUC_TEST(), then in
> +the main() ZUC_ADD_TEST() should be called for each defined test. Finally
> +ZUC_RUN_TESTS() should be called.

That seems a bit more complicated, having to write the test name
twice, in both ZUC_TEST and ZUC_ADD_TEST. Our existing approach
needs only the TEST part.

> +Tests can use several ZUC_EXPECT_* or ZUC_ASSERT_* checks to validate
> +conditions. The ZUC_EXPECT_* ones upon failure will mark the current test
> +as failing, but allow execution to continue. On the other hand, the
> +ZUC_ASSERT_* tests will mark the current test as failed and then immediately
> +terminate the test.
> +
> +The set of non-fatal test checks are
> +
> +- ZUC_EXPECT_TRUE()
> +- ZUC_EXPECT_FALSE()
> +- ZUC_EXPECT_EQ()
> +- ZUC_EXPECT_NE()
> +- ZUC_EXPECT_LT()
> +- ZUC_EXPECT_LE()
> +- ZUC_EXPECT_GT()
> +- ZUC_EXPECT_GE()
> +
> +The set of fatal test checks are
> +
> +- ZUC_ASSERT_TRUE()
> +- ZUC_ASSERT_FALSE()
> +- ZUC_ASSERT_EQ()
> +- ZUC_ASSERT_NE()
> +- ZUC_ASSERT_LT()
> +- ZUC_ASSERT_LE()
> +- ZUC_ASSERT_GT()
> +- ZUC_ASSERT_GE()
> +
> +*/

These EXPECT and ASSERT macros are what you have better than the
existing testing framework. These macros can actually print the
values they compare, not just the condition like assert() does.
There's also the difference between fatal and non-fatal failures.

The other merit is that you document stuff. That's always good.

But, after skimming through the whole patch, I am left with the
question: why should we replace our ad hoc framework with this ad
hoc framework? What benefits does this have over the old? That is
the thing I couldn't see.

What I saw was just reimplementing what our old framework already
does. In some cases better (EXPECT and ASSERT instead of just
assert()) and in many cases worse (e.g. not forking a process for
each test, which means a crashing test takes the whole test harness
down).

I'm left with the feeling that it would be better to improve the
existing framework: fix its shortcomings, document it, and keep the
good bits.


Thanks,
pq