probabilistic approach to tinderboxing

[Bjoern Michaelsen]

Hi,

On Thu, Jun 14, 2012 at 12:36:12PM +0200, Lubos Lunak wrote:
>  Where are we going to get these tinderboxes? The page for master lists 17 
> tinderboxes, out of which only about 11 build regularly. Out of which there 
> are 4 linux-gcc tinderboxes, one linux-clang, 1(2?) macosx, 3 windows and 1 
> mingw. That, not counting any further division like running or not running 
> checks, gives 5 testcases if I understand it correctly, and building 
> successfully in any of them does not guarantee that the rest builds as well.

Right -- although I would count the linux-gcc-with-subsequentcheck tinderbox as
a single test case.

>  So, the thing I do not understand, how is this (complex, as far as I can say) 
> setup supposed to improve anything? There does not seem to be any big 
> difference between dedicating X tinderboxes to it or just 1-2 tinderboxes. If 
> we put a number of tinderboxes on this, they still won't test the patches 
> enough, while there won't be that many left for checking that the actual 
> repository builds fine, which will be needed anyway.

I dont think we should split up tinderboxes between this and repository builds
(I asssume you mean testbuilding the head of master with that). As the patches
are cherrypicked on the head of master with this, master is already tested
along with this. However, while people are in the beginning still pushing even
risky patches directly to master we cant assume master to be flawless all of
the time -- so the proposed algo should be extended with "build the master you
cherry-picked upon once after each failed build".

What is this setup improving? Simple:
- find buildbreakers _before_ they hit master
- with gerrit, you can push your patch, wait for the tinderbox result you need
  and _then_ put it on master
  (so this gives everyone easy access to windows test builders, which as we all
  know are a pain to set up, and also subsequentcheck runs)
- reduce tinderbox spammage as the list of possible offenders is smaller -- and
  esp. is not growing with every build
- allows greenlighting bigger sets of patches with one compile (essential on
  slower platforms, f.e. Windows)

>  If patches in gerrit[*] need to be tested, it should work just as well to use 
> one of the really fast tinderboxes for checking that it builds, which will 
> cover a majority of possible problems, then maybe one more can build 
> including checks, and if any problem gets through, it'll be caught by the 
> normal tinderboxes, which are needed anyway (for direct commits, some other 
> seemingly unrelated commit breaking the patch meanwhile, etc.).

That doesnt scale that well. Also note that the tinderboxes building plain
master should end up a scenario that is very, very rarely needed (it will still
be done of course)

Best,

Bjoern

> [*] Speaking of which, when will we get told about this gerrit thing, besides 
> random remarks, or have I missed it?

Look for a mail with "ACTION REQUIRED" in the subject in the libreoffice
mailing list archive.