[PATCH weston 1/5] tests: always build tests

[Sam Spilsbury]

On Wed, Sep 11, 2013 at 8:38 PM,  <sardemff7+wayland at sardemff7.net> wrote:
> On 11/09/2013 11:12, Sam Spilsbury wrote:
>>
>> Quick thought: there's also an important psychological effect to
>> building the tests on a standard make because it promotes them to the
>> same importance as the rest of your code. They become less of an
>> afterthought and it promotes greater care around how people design
>> the tests (eg, making the tests clean, making sure they run
>> quickly), as well as how the rest of the codebase interacts with the
>> tests. We observed a similar effect at Canonical between the projects
>> which had test building on by default as opposed to those that did
>> not.
>
>
> Then we should definitely fix users (developers) and their workflow,
> not some arbitrary “problem”, as I said already.

I think we're all in agreement that having maintained tests is a good thing.

Generally speaking I've found that most projects which use a
test-driven workflow have them *built* by default and then have a
separate rule (eg make check or make test) to run them. They also have
a CI system which automatically runs the tests and fails patches which
fail the tests. I don't know how well CI systems would work for a
patches-on-the-mailinglist type workflow, but there are plenty of very
well integrated systems for code hosts like launchpad and github.

Building tests by default makes them more obvious, especially to
newcomers. Running them by default would certainly make them even more
obvious. But we don't run them by default because then we would have
to accept the constant time process of running the tests every time
"make" is invoked (which is about seven seconds in our case). There's
also the problem with the weston tests not running headless (although
they should). A run of the tests isn't mandatory every time the build
completes - its up to the discretion of the developer as to when it
would be appropriate to run them. Building them by default at least
gives you quick feedback of when you stepped on a test.

Building the tests by default is a structural change that promotes
positive impacts on workflow and more thought about testing. It isn't
an entire solution to the problem of developers not writing tests, but
it certainly gets you part of the way there. At least with a testsuite
which has no dependencies, there doesn't seem to be a very good reason
not to at least build them by default.

Best Regards,

Sam.


-- 
Sam Spilsbury