test-infra proposal: master-tested branch

[Bjoern Michaelsen]

Hi,

On Wed, Jun 10, 2015 at 10:02:12PM +0100, Wols Lists wrote:
> Would it make sense to have a server dedicated to Heisenbugs?
> If a test triggers a heisenbug, disable it on most of them 

That makes sense -- thus the suggestion to have 'master-tested' run e.g. make
check multiple -- say 10 times. If there is a new Heisenbug affecting a test,
it will be disabled and thus will not hurt CI testing.

> ... but try to
> instrument the heisenbug tester up the wazoo so that when it fails, there's a
> pile of logs to try and work out what went wrong.

In theory yes, in practice hunting most Heisenbugs is not too effective. For
example, we still have over 300 bisected regressions. And quite a few of them
will have the same root cause as some Heisenbug: However, in general it is much
easier to fix a well triaged regression than cutting through the haze and
finding the one precious hint hidden in piles and piles of logs.

Best,

Bjoern

P.S.:
If the Heisenbug is a regression and half-way reliable reproducable (say
reliable failure in ten runs) it should be (bi)bisectable. It then depends on
the test: If its a test using pure UNO, it should be possible to bibisect
without rebuilding LibreOffice on every step. If it is using more than UNO,
bibisecting wont work and one has to build LibreOffice on every step.