'make check' fails when building with gcov code coverage

[Stephan Bergmann]

On 31/05/2022 13:43, Maarten Hoes wrote:
> Anyway, if more tests fail sometimes without there being a 'real' 
> failure, then I am not sure how to deal with that in relation to 
> generating an lcov report for a full build. My initial idea was to not 
> generate a report if 'make check' failed, but hearing this now makes me 
> wonder if that would be a good approach. Perhaps it would be preferred 
> just to run 'make -k check' and generate a report always, regardless of 
> test failures ? Of course you could also choose to skip such tests, but 
> that would lead to not representative results; and someone would have to 
> manually keep the skip list up to date, which people will forget to do.

Occasionally failing tests are a well-known problem for LO (e.g., 
witness the "Jenkins / CI update: tests that failed more than twice in 
last seven days" section in the weekly ESC minutes---many of those are 
apparently spuriously failing tests), and there is no reason to assume 
that your gcov builds would not also occasionally be affected by that.

So the general advice would be to ignore occasional failed builds (which 
might not only fail due to spurious test failures, but also because e.g. 
a build breaker got submitted by accident).  If some specific tests 
cause enough builds to fail to make that approach impractical, those 
tests should get fixed.  Or, as a last resort, get disabled for 
known-failing build scenarios.

I don't think -k would be a good solution, as it would make it harder to 
meaningfully interpret the generated data.