[waffle] CI, static analisys, tests and waffle

[Chad Versace]

On Tue 01 Sep 2015, Emil Velikov wrote:
> Hello all,
> 
> Not too long ago I had the crazy idea of using some of the free
> utilities out there with waffle.
> 
> For example Coverity provides a free scan to open-source projects,
> which nicely integrates with Travis-CI,
> The latter of which supporting Linux and MacOS, while an alternative
> is available for Windows (AppVeyor).
> 
> I Travis + Linux + Coverity a try and things turned out ok. Small
> catch is the wayland functionality tests fail, as they run in a VM,
> while (the old) mesa/wayland requires hardware device.
> We can split these into separate gl_basic_foo_test (foo being
> glx,wayland etc.) or just silence the whole lot.

Running Linux + Coverity in CI system is a good idea.

As for Waffle's functional tests, we should separate them out so it's
possible to run tests for individual platforms. For example, run the GBM
tests but not the Wayland tests. Then we can configure the CI to run
just the tests that will actually work.

Being able to run the GBM tests in CI should find real bugs. As Mark
pointed out, Mesa commits have broken Waffle + Piglit before, and GBM
tests would have caught that.

> I have also moved the gl_basic_test to cmocka (after updating the
> latter to 1.0.1) and nuked waffle_test. With this each test has it's
> own setup/teardown and things are handled correctly, fixing memory
> leaks.

Nice. I look forward to the replacement of waffle_test with cmocka.

> Overall I'm soliciting feedback on the topic(s) as some of the patches
> are rather rough and/or complete.
> Do you think that these are worthy additions and which one would you
> like to see first :-)

Yes, I think this is worth doing. I prefer to see the cleanups arrive in
this order, but it's not a strong preference:

    *. Fix bugs and leaks as they're discovered.
    1. Replace waffle_test with cmocka.
    2. Split `make check-func` into a separate target for each platform.
    3. Add `make check` and the GBM tests to some CI system.