CppunitTest: different results depending on execution command

[Miklos Vajna]

Hi Xisco,

On Fri, Jul 24, 2020 at 07:51:30PM +0200, Xisco Fauli <xiscofauli at libreoffice.org> wrote:
> as I mentioned yesterday in the ESC meeting, i'm working on a unittest
> that is passing for me locally but it fails in Jenkins. The patch is
> here: https://gerrit.libreoffice.org/c/core/+/99214
> 
> it turned out it passes when executed with 'make CppunitTest_sw_uiwriter
> CPPUNIT_TEST_NAME="testTdf132911::TestBody"' but fails when executed
> with 'make CppunitTest_sw_uiwriter'.
> 
> Any idea why it passes when the test is executed alone and why it fails
> when not ?

Most probably one of the tests violate that invariant that none of the
test cases in a suite should alter the global state. You could try to
disable the tests from the suite, i.e. binary search what test causes
the problem. Hopefully you can find a single 2nd test that makes your
new test fail, and then that's easier to investigate that the whole
sw_uiwriter suite, which has 300+ test cases.

(Somewhat unrelated, but more, smaller test suites also help with
parallelism, and are easier to debug when they fail. So while it's
interesting to see what is the root cause in the above case, you may as
well just put your new test to sw/qa/core/doc/doc.cxx. Like, the fix was
in DocumentRedlineManager ->
sw/source/core/doc/DocumentRedlineManager.cxx -> matching smaller suite
is sw_core_doc.)

Regards,

Miklos