git and Marge troubles this week
emma at anholt.net
Fri Jan 7 05:36:34 UTC 2022
As you've probably noticed, there have been issues with git access
this week. The fd.o sysadmins are desperately trying to stay on
vacation because they do deserve a break, but have still been working
on the problem and a couple of solutions haven't worked out yet.
Hopefully we'll have some news soon.
Due to these ongoing git timeouts, our CI runners have been getting
bogged down with stalled jobs and causing a lot of spurious failures
where the pipeline doesn't get all its jobs assigned to runners before
Marge gives up. Today, I asked daniels to bump Marge's pipeline
timeout to 4 hours (up from 1). To get MRs flowing at a similar rate
despite the longer total pipeline times, we also enabled batch mode as
described at https://github.com/smarkets/marge-bot/blob/master/README.md#batching-merge-requests.
It means there are now theoretical cases as described in the README
where Marge might merge a set of code that leaves main broken.
However, those cases are pretty obscure, and I expect that failure
rate to be much lower than the existing "you can merge flaky code"
failure rate and worth the risk.
Hopefully this gets us all productive again.
More information about the mesa-dev