Broken master and newcomers.

Wed Apr 6 22:22:01 UTC 2016

On Wed, Apr 6, 2016 at 1:04 AM, jan iversen <jancasacondor at gmail.com> wrote:
> Hi.
>
> I experience quite frequently that new contributors follow the instructions, but end up with a master that will not compile,

these day that is unlucky:

here are the stats for the last 14 days:

    from:Wed Mar 23 21:30:24 2016
    master linux rel  jobs: 296 ok: 282 ko:  11 fail ratio:  3.72 %
break:   9 broken duration: 2.12%
    master linux dbg  jobs: 242 ok: 226 ko:  15 fail ratio:  6.20 %
break:  12 broken duration: 2.98%
    master mac rel    jobs: 281 ok: 274 ko:   7 fail ratio:  2.49 %
break:   4 broken duration: 0.86%
    master mac dbg    jobs: 285 ok: 282 ko:   3 fail ratio:  1.05 %
break:   2 broken duration: 1.95%
    master win rel    jobs: 222 ok: 212 ko:  10 fail ratio:  4.50 %
break:   4 broken duration: 1.00%
    master win dbg    jobs: 229 ok: 215 ko:  14 fail ratio:  6.11 %
break:   7 broken duration: 2.64%
    master win64 dbg  jobs: 232 ok: 216 ko:  15 fail ratio:  6.47 %
break:   7 broken duration: 3.19%

broken duration is giving you the following info:
if you randomly pull master at any time in the period considered: what
was the odd that it was broken at that time
during the last 14 days your odd varied from 1 in a 115 for mac
releases build to 1 in 30 for windows64 dbg

let's look at windows dbg:

the last break right this minute on windows dbg was 1 day 9 hours ago
http://ci.libreoffice.org/job/lo_tb_master_win_dbg/5935/
"Device or resource busy"
either a parallelism problem in our build system or a bug in Windows.
master was not really red because of that.

the previous break was
http://ci.libreoffice.org/job/lo_tb_master_win_dbg/5915/
2 days 5 hours ago.. direct push by sberg

the one before was 2 hour earlier
http://ci.libreoffice.org/job/lo_tb_master_win_dbg/5913/

that one seems attributable to Windows crappy FS.

the one before that was
http://ci.libreoffice.org/job/lo_tb_master_win_dbg/5878/
5 days 13 hours ago, this one failed on win-dbg tb but passed Jenkins.

the one before that was 6 days 11 hours ago
http://ci.libreoffice.org/job/lo_tb_master_win_dbg/5851/
that was due to 41bdaa37cc62f656cc164992c4c7d39bec7e57e2 a direct push by noel

the one before that was 7 days 16 hr ago
and was not due to the commit.. but some jenkins <-> windows interaction problem
Note that these kind of 'break' would not actually impact someone
pulling master.
iow it is red in ci but not really red

so of the win dbg break... 3 where due to windows glitches. 1 was
properly vetted by Jenkins but still failed in that tinderbox
2 could have been avoided by using gerrit.

but in the end that is 3 real break in 2 weeks for a real red duration
of 2 hours 38 minutes that is 1 chance in 127 to get a random red
state.

so :"I experience quite frequently " seems to be a 'count the
his/ignore the misses' kind of thing.

IOW: my advice... let's keep nudging people to avail themselves of the
CI facility and avoid direct push...
that would have change the odd of a random red pull from 1 to 127 to 1
in 305 in this particular case.
but let's not forget that not that long ago it was easier to count how
often windows was buildable rather than hunt the commit where it was
not.

I also have another data point: the bibisect build: these are less
sensitive since I purposefully build them without Werror and I
purposfully do not run the checks
(the goal being to have binaries.. not to validate the build quality)

here are the result: this are the commit in bibisect that cover more
than 1 source commit. the number at the end of each line is the number
of unbuildable commit that are folded in that one bibisect commit

1dee09a4aa086524fd7704ad3dd6dc7e9ba201f4:1
1ebeb8fb59bfff5b9bdb3ac6db158c7c13e85e70:1
2a019e2b222ad49446c730bdbdea6c6651e36d91:1
3229e0f25f5d55d566208e42b276339a9c3c63c5:1
4c4af51cf6105e635feb206ea193f150f05e0edf:1
6d6a571cd180ee828ec8c8a5bd21a46d571d47ee:1
81168aee40fd18d7fcc475406f4c58ff7ba63455:1
8a70721672f5a1bce10547849435d7a0e286fb45:1
a54383ef92868298a368bcec871e5f2f2b2343b6:1
a9cca36433706038da0a51d6311da1bb77eb6112:1
e5c3bed99f2508b10833ccada394ce1abc37bf07:1
e72fd72eaea986ba41b8afff31ae9b77177fc685:1
f3ed988856fe00c5826781d97aaf2cae61f1abb7:1
4498e0cb3277ce65738acbc7c1faac43e9066430:2
6d46b870384fbfce0293a8fce41e373087ff681f:2
7b7852a9877e1123c60d5872ecf8ff0d78da45d7:2
ac5a57d70ad872f2dfe9d1bff73baa5318042e0e:2
b89bf27413eaf5cfab9a501a297756b8ef22eaa2:2
c6f1dc4f2407945334adef66c989f3878abc7043:2
cc822d9c6af1e8d6e937e23145471daf52ebd472:2
f73273c14261004eae364d06509b029c3a3883f2:2
45efc63eb004ebd2643cea9ebcdd810d0935bfa6:3
5de2c9cdc2e246e98ab43dde0baf14ca0947dd47:3
90fbb1101724c057d3c269f3b05f5584f701bb2f:3
b1d96f0e8acd28c6536896ed9aa70a88daa3f8fc:3
ba10b24c5f679cfcace557a920c83264b374e054:4
cebf63930f184875b7a04b3c95ad182554df5aa6:4
1d25fbbb38f0973147eda92bde66872b7cc974c0:5
1cb794851d2cbb56563a82b763da6c8a696dd465:6
326d9741e8236d3912ca55b0e9a7d4cee7b5a882:6
9e601b0af6620629a4aaf739551809b122cc55a2:7
f7a38779df3d655b6f19db543a2a6fb938d008db:7
a95751670b0782a3beff38628e8dfef9b5f6b843:8
307d0ab9e2227c665679eb7c63d8eaff000c6e7a:9
46360ca22a8c3fd994d47e131c35b119d4c53564:10
4cb47fa02103f02cc229a09855da15892018cd5f:10
2addb6810f00cbf802a6e5be52192f0957f2da50:11
95f3f2c574dd467a50abbc5531ccf9545ccf21cc:12
a43223601d935530b269e8ff9d9a0e673d021141:12
2f5ea0f9d3fe16cf246ccaac2e44b41de9f701f4:14
e3fd7ed5afe94a7c4c442c4990f3cc7133912a90:14
3acad46ef229f3bd1b2fe9399af7e3b5bf6b6d33:15
a4766b0b167dce9be0537040f3306c9cfc0606ec:18
dfc2a983418964c4608f8f7b23d9947cdea7a1d6:22
60382473d6b952978ed66b2c63db328cab04d4a1:25
d34dbc4155245e76b72e617b7e8123e29b4055ed:29
74b9a0610d51bbd4dd789766e42ebcda8ea810dc:37
31c7ac9f3bb171a85cbf0d2f5c506de765ebcd66:79

that is 405 commit in 48 break.
that is out of 5405 bibisect commit, representing 5810 source commit
there was 405 commit that would not compile and link

the 3 worse streak representing 79, 37 and 29 commit respectively all
occurred very early in the epoch (more than 2 month ago)

Finally there is a fairly easy way around this problem:

http://tinderbox.libreoffice.org/MASTER/status.html
when you think about pulling go there and check the state of master...
if you see some red scroll down find a green point and get the sha of that point

clone and checkout a branch at that sha...

Norbert