[PATCH 0/3] Resubmit - Unit test framework for Wayland

[Kristian Høgsberg]

On Fri, Mar 2, 2012 at 4:36 AM, Michael Hasselmann
<michaelh at openismus.com> wrote:
> On Thu, 2012-03-01 at 22:34 -0500, Kristian Høgsberg wrote:
>> Hi Artie,
>>
>> Thanks for starting this.  Looks good and certainly when we start
>> adding tests for some of the more complex objects and data structures
>> in the library (wl_map would be a good next step), it will be a good
>> way to avoid regressing functionality.  I'm not convinced that we
>> really need an external unit testing framework though. All each TESTS
>> binary need to do is to test something and fail or succeed.  We can
>> add a little test helper to provide the fail_if() etc functions.
>
> I found that a good testing framework can lower the barrier of writing
> useful tests. Nice logging and status reports are important I feel. And
> if for example you can easily write data driven tests, then testing all
> possible code paths in a critical area becomes straight-forward and the
> tests will remain readable (and with that, maintainable; people tend
> forget that every test also adds to the overall maintenance costs which
> can often enough outweight the benefits of the test). It's actually the
> one thing I really like about Qt's testlib.

I'm just not convinced that check does any of that.  It's an extra
dependency on a little-known library for a project that's already too
hard to build for many people.  All I see in the patch is that the
test case code flow is obfuscated by setting up test-suites and other
boiler plate.  Keeping the barrier to writing test cases low to me
means that is should be transparent and obvious what's going on and
that the required boiler plate code is kept to a minimum.  So no, not
convinced.

Kristian

> If a testing framework even goes so far to allow fuzz testing out of the
> box, then you have a nice sanity check for your APIs.
>
> I have no experience in writing a testlib, but I would assume it's a
> non-trivial task.

If you're writing a do-it-all, general purpose test framework, perhaps
(check is only 2000 lines of code though).  If we just organically
grow our test helpers as we need new functionality, not so much.

Kristian