git and Marge troubles this week

[Ian Romanick]

Blarg. That all sounds awful.  I think (hope!) I speak for everyone when 
I say that we all appreciate your and daniels' efforts to keep this big 
piece of machinery working.

If the problems persist much longer, should we consider pushing out the 
22.0 branch point?

On 1/6/22 9:36 PM, Emma Anholt wrote:
> As you've probably noticed, there have been issues with git access
> this week.  The fd.o sysadmins are desperately trying to stay on
> vacation because they do deserve a break, but have still been working
> on the problem and a couple of solutions haven't worked out yet.
> Hopefully we'll have some news soon.
> 
> Due to these ongoing git timeouts, our CI runners have been getting
> bogged down with stalled jobs and causing a lot of spurious failures
> where the pipeline doesn't get all its jobs assigned to runners before
> Marge gives up.  Today, I asked daniels to bump Marge's pipeline
> timeout to 4 hours (up from 1).  To get MRs flowing at a similar rate
> despite the longer total pipeline times, we also enabled batch mode as
> described at https://github.com/smarkets/marge-bot/blob/master/README.md#batching-merge-requests.
> 
> It means there are now theoretical cases as described in the README
> where Marge might merge a set of code that leaves main broken.
> However, those cases are pretty obscure, and I expect that failure
> rate to be much lower than the existing "you can merge flaky code"
> failure rate and worth the risk.
> 
> Hopefully this gets us all productive again.