[waffle] CI, static analisys, tests and waffle

[Emil Velikov]

On 3 September 2015 at 21:27, Chad Versace <chad.versace at intel.com> wrote:
> On Tue 01 Sep 2015, Emil Velikov wrote:
>> Hello all,
>>
>> Not too long ago I had the crazy idea of using some of the free
>> utilities out there with waffle.
>>
>> For example Coverity provides a free scan to open-source projects,
>> which nicely integrates with Travis-CI,
>> The latter of which supporting Linux and MacOS, while an alternative
>> is available for Windows (AppVeyor).
>>
>> I Travis + Linux + Coverity a try and things turned out ok. Small
>> catch is the wayland functionality tests fail, as they run in a VM,
>> while (the old) mesa/wayland requires hardware device.
>> We can split these into separate gl_basic_foo_test (foo being
>> glx,wayland etc.) or just silence the whole lot.
>
> Running Linux + Coverity in CI system is a good idea.
>
> As for Waffle's functional tests, we should separate them out so it's
> possible to run tests for individual platforms. For example, run the GBM
> tests but not the Wayland tests. Then we can configure the CI to run
> just the tests that will actually work.
>
> Being able to run the GBM tests in CI should find real bugs. As Mark
> pointed out, Mesa commits have broken Waffle + Piglit before, and GBM
> tests would have caught that.
>
>> I have also moved the gl_basic_test to cmocka (after updating the
>> latter to 1.0.1) and nuked waffle_test. With this each test has it's
>> own setup/teardown and things are handled correctly, fixing memory
>> leaks.
>
> Nice. I look forward to the replacement of waffle_test with cmocka.
>
>> Overall I'm soliciting feedback on the topic(s) as some of the patches
>> are rather rough and/or complete.
>> Do you think that these are worthy additions and which one would you
>> like to see first :-)
>
> Yes, I think this is worth doing. I prefer to see the cleanups arrive in
> this order, but it's not a strong preference:
>
>     *. Fix bugs and leaks as they're discovered.
>     1. Replace waffle_test with cmocka.
>     2. Split `make check-func` into a separate target for each platform.
Considering the churn during the conversion, we might want to do it in
one step, namely:
1. Update cmocka (fix integration, add missing include)
2. Update existing users to cmocka 1.x
3. Transition/refactor each platform to a separate cmocka based test
`gl_basic_test_$(platform)'

>     3. Add `make check` and the GBM tests to some CI system.
I don't see any GBM tests in waffle. Am I missing something or you
have something locally ?

Thanks
Emil