'make check' fails when building with gcov code coverage

[Maarten Hoes]

Hi,

On Tue, May 31, 2022 at 2:07 PM Stephan Bergmann <sbergman at redhat.com>
wrote:

> So the general advice would be to ignore occasional failed builds (which
> might not only fail due to spurious test failures, but also because e.g.
> a build breaker got submitted by accident).  If some specific tests
> cause enough builds to fail to make that approach impractical, those
> tests should get fixed.  Or, as a last resort, get disabled for
> known-failing build scenarios.
>
> I don't think -k would be a good solution, as it would make it harder to
> meaningfully interpret the generated data.
>

Ok, so let me see if I understood that correctly :

*if* lcov reports were to be generated automatically on a regular basis
again, then the desired behaviour would be to just let the build fail/stop
(whatever the reason, gcov related, spuriously failure, developer mistake),
don't generate a lcov report when it does fail, and if it fails often
enough within a certain timeframe people will investigate and determine
whether the failing check should be fixed or disabled, and a new gcov/lcov
report should only be generated if the build and 'make check' both succeed.




> Occasionally failing tests are a well-known problem for LO (e.g.,
> witness the "Jenkins / CI update: tests that failed more than twice in
> last seven days" section in the weekly ESC minutes---many of those are
> apparently spuriously failing tests), and there is no reason to assume
> that your gcov builds would not also occasionally be affected by that.
>

Although I realize - as you explained - that there are occasional test
failures for all Jenkins builds, I am also not sure if this is what is
going on here specifically. In order to better determine if we are dealing
with occasional or structurally failing tests for the mentioned 3 tests
specifically (UITest_solver, CppunitTest_sccomp_solver,
CppunitTest_sccomp_swarmsolvertest), I did some more testing (fresh git
pull and build), and when I do a gcov build and make check, these 3 fail
for me reliably (and they succeed for a non-gcov 'regular' build). All the
other tests succeed, but these 3 keep failing for me. I ran them multiple
times in a row, and they keep failing for each run. So while I realise that
this might not be 100% proof, I also get the strong impression that for
these 3 specifically, we are not talking about 'occasionally' failing
tests, but about structurally failing tests when run on a gcov build, and
this needs to be looked into. Of course, the fact that they fail reliably
for me on every run I tried says nothing about what makes that so: for
example, it can still be timeout related, as I can imagine (but did not
test) a gcov enabled test taking longer to complete than a test run on a
'regular' build. The reason that I keep going on about this is not because
I am purposefully trying to be annoying, but rather because I personally
strongly think/feel (but I may be mistaken) that in order for people to be
interested in reviving automated lcov reports at all, it first needs to be
made sure that the automation works 'most of the time' (or at least as much
as the other Jenkins builds), and not fail on every single run right at the
start.



- Maarten
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/libreoffice/attachments/20220601/b19b2202/attachment.htm>