test-infra proposal: master-tested branch

[Wols Lists]

On 10/06/15 20:22, Norbert Thiebaud wrote:
> - a test auto-induced one: when a test is unstable and produce random
> failures based on circumstances... the infamous 'heisenbugs'
> and heisenbug can be a systemic/design problem or can be a real bug
> that is hard to trigger. either way these are not useful, and in fact
> harmful in a ci context; because the human nature is 'If you can't
> reproduce it is not a bug'
> so the later category of real hard to trigger bug is always labeled
> 'systemic error' and ignored anyway... and it make people numb to
> errors...
> For automated testing, trust is paramount: heisenbug test failure are
> the enemy, false non-failure is bad but actually less painful

Would it make sense to have a server dedicated to Heisenbugs? If a test
triggers a heisenbug, disable it on most of them but try to instrument
the heisenbug tester up the wazoo so that when it fails, there's a pile
of logs to try and work out what went wrong.

And of course, it doesn't spam failure reports generally unless someone
asks for them (the person to whom the bug is assigned should get them,
of course), but it does save those logs for forensic analysis.

Cheers,
Wol