[PATCH weston] Added simple unit/integration test framework and corresponding test program.

[Jon A. Cruz]

On 05/04/2015 12:22 PM, Pekka Paalanen wrote:
> On Wed, 29 Apr 2015 15:31:45 -0700
> "Jon A. Cruz" <jonc at osg.samsung.com> wrote:
>
>> Added a simple C-based test framework and an example program
>> that uses it to run through some simple wayland client checks.
>>
>
> Hi Jon,
>
> I said I didn't have time to look at this, but I decided to take a
> peek as "bed time reading", just to see what you were up to.
>
> It seems that all this patch does is duplicate perhaps half
> of what our ad hoc testing framework already does. It is not
> without merits though, more on that below.
>

[SNIP]

>
> Doesn't seem too different from what we had already...
>
>> +To get started, one or more tests should be defined via ZUC_TEST(),
then in
>> +the main() ZUC_ADD_TEST() should be called for each defined test.
Finally
>> +ZUC_RUN_TESTS() should be called.
>
> That seems a bit more complicated, having to write the test name
> twice, in both ZUC_TEST and ZUC_ADD_TEST. Our existing approach
> needs only the TEST part.
>

Yes, it was requiring two steps since it was the minimal implementation
using portable C. Since we are depending on gcc anyway I went ahead and
used compiler extensions to get it to a single line.

>
> These EXPECT and ASSERT macros are what you have better than the
> existing testing framework. These macros can actually print the
> values they compare, not just the condition like assert() does.
> There's also the difference between fatal and non-fatal failures.
>
> The other merit is that you document stuff. That's always good.
>
> But, after skimming through the whole patch, I am left with the
> question: why should we replace our ad hoc framework with this ad
> hoc framework? What benefits does this have over the old? That is
> the thing I couldn't see.
>
> What I saw was just reimplementing what our old framework already
> does. In some cases better (EXPECT and ASSERT instead of just
> assert()) and in many cases worse (e.g. not forking a process for
> each test, which means a crashing test takes the whole test harness
> down).
>
> I'm left with the feeling that it would be better to improve the
> existing framework: fix its shortcomings, document it, and keep the
> good bits.
>

Good.

My main intent in getting this submitted now was to verify which
direction things should go, and which aspects to focus on. The test
runner implementation was bare minimum, with most of the time on setting
up the test macros themselves. My other design goal here was to
explicitly separate the different layers that are in play in order to
clean things up for faster use. The newer 'waycheck' binary should work
against any compositor, so it needs to avoid any of the weston-specific
bits.

I'll move forward with merging things together. At the moment there is
some general testing work that others have outstanding, so I'll hold off
on modifying the existing files. Most likely some will just take
migrating appropriate functionality to the separated files.

One thing I have pending on docs are quick diagrams of the current code
organization and a proposed end state. Those should be sent out soon for
additional info and review.


-- Jon A. Cruz - Senior Open Source Developer
Samsung Open Source Group
jonc at osg.samsung.com