test-infra proposal: master-tested branch

[Michael Stahl]

On 10.06.2015 23:36, Bjoern Michaelsen wrote:
> On Wed, Jun 10, 2015 at 10:02:12PM +0100, Wols Lists wrote:
>> Would it make sense to have a server dedicated to Heisenbugs?
>> If a test triggers a heisenbug, disable it on most of them 
>> ... but try to
>> instrument the heisenbug tester up the wazoo so that when it fails, there's a
>> pile of logs to try and work out what went wrong.
> 
> In theory yes, in practice hunting most Heisenbugs is not too effective. For
> example, we still have over 300 bisected regressions. And quite a few of them
> will have the same root cause as some Heisenbug: However, in general it is much
> easier to fix a well triaged regression than cutting through the haze and
> finding the one precious hint hidden in piles and piles of logs.

it would perhaps be easier to track down Heisenbugs in tests with
http://rr-project.org/

the tool has some limitations such as
- only works on Linux x86/amd64
- the 3.2.0 release doesn't work with LO but current master build does
- binaries have to be *identical*, so if you run it on a tinderbox
  you have to stop it until the heisenbug is debugged
- due to the last point you can't add debug logging code after-the-fact,
  you'd have to record again until you hit the bug again to get the log
- gdb conditional breakpoints are *really* slow if you set one in a
  location that is hit millions of times

... but determinism is nice when it works, and you can already record
all JUnit based tests with "RR=1 make check" on master.