[PATCH 0/3] Resubmit - Unit test framework for Wayland

[Eoff, Ullysses A]

>-----Original Message-----
>From: hoegsberg at gmail.com [mailto:hoegsberg at gmail.com] On Behalf Of
>Kristian Høgsberg
>Sent: Friday, March 02, 2012 7:33 AM
>To: Michael Hasselmann
>Cc: Eoff, Ullysses A; wayland-devel at lists.freedesktop.org
>Subject: Re: [PATCH 0/3] Resubmit - Unit test framework for Wayland
>
>On Fri, Mar 2, 2012 at 4:36 AM, Michael Hasselmann
><michaelh at openismus.com> wrote:
>> On Thu, 2012-03-01 at 22:34 -0500, Kristian Høgsberg wrote:
>>> Hi Artie,
>>>
>>> Thanks for starting this.  Looks good and certainly when we start
>>> adding tests for some of the more complex objects and data structures
>>> in the library (wl_map would be a good next step), it will be a good
>>> way to avoid regressing functionality.  I'm not convinced that we
>>> really need an external unit testing framework though. All each TESTS
>>> binary need to do is to test something and fail or succeed.  We can
>>> add a little test helper to provide the fail_if() etc functions.
>>
>> I found that a good testing framework can lower the barrier of writing
>> useful tests. Nice logging and status reports are important I feel. And
>> if for example you can easily write data driven tests, then testing all
>> possible code paths in a critical area becomes straight-forward and the
>> tests will remain readable (and with that, maintainable; people tend
>> forget that every test also adds to the overall maintenance costs which
>> can often enough outweight the benefits of the test). It's actually the
>> one thing I really like about Qt's testlib.
>
>I'm just not convinced that check does any of that.  It's an extra
>dependency on a little-known library for a project that's already too
>hard to build for many people.

Check is used successfully in quite a few projects (gstreamer, gnupdf, ...).
It's also available through several major distro package managers (Fedora,
Ubuntu, ...).  So I don't think it's a "little-known" library or hard to get.

>All I see in the patch is that the
>test case code flow is obfuscated by setting up test-suites and other
>boiler plate.  Keeping the barrier to writing test cases low to me
>means that is should be transparent and obvious what's going on and
>that the required boiler plate code is kept to a minimum.  So no, not
>convinced.
>

It wasn't obvious or transparent to write tests?  How so?

>Kristian
>
>> If a testing framework even goes so far to allow fuzz testing out of the
>> box, then you have a nice sanity check for your APIs.
>>
>> I have no experience in writing a testlib, but I would assume it's a
>> non-trivial task.
>
>If you're writing a do-it-all, general purpose test framework, perhaps
>(check is only 2000 lines of code though).  If we just organically
>grow our test helpers as we need new functionality, not so much.
>

I think growing our own framework will be more maintenance in the
long run.  Check is small yet provides a lot of useful functionality that
is abstracted away from developers so they can focus on writing tests
instead of all the gory details of setting them up.

If anyone knows of a better existing unit test framework, then please
share.

Artie

>Kristian