<div dir="ltr">We might want to consider pushing out the branch point a week anyway to help people get CTS in order?</div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 7, 2022 at 1:08 PM Ian Romanick <<a href="mailto:idr@freedesktop.org">idr@freedesktop.org</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Blarg. That all sounds awful. I think (hope!) I speak for everyone when I say that we all appreciate your and daniels' efforts to keep this big piece of machinery working. If the problems persist much longer, should we consider pushing out the 22.0 branch point? On 1/6/22 9:36 PM, Emma Anholt wrote: > As you've probably noticed, there have been issues with git access > this week. The fd.o sysadmins are desperately trying to stay on > vacation because they do deserve a break, but have still been working > on the problem and a couple of solutions haven't worked out yet. > Hopefully we'll have some news soon. > > Due to these ongoing git timeouts, our CI runners have been getting > bogged down with stalled jobs and causing a lot of spurious failures > where the pipeline doesn't get all its jobs assigned to runners before > Marge gives up. Today, I asked daniels to bump Marge's pipeline > timeout to 4 hours (up from 1). To get MRs flowing at a similar rate > despite the longer total pipeline times, we also enabled batch mode as > described at <a href="https://github.com/smarkets/marge-bot/blob/master/README.md#batching-merge-requests" rel="noreferrer" target="_blank">https://github.com/smarkets/marge-bot/blob/master/README.md#batching-merge-requests</a>. > > It means there are now theoretical cases as described in the README > where Marge might merge a set of code that leaves main broken. > However, those cases are pretty obscure, and I expect that failure > rate to be much lower than the existing "you can merge flaky code" > failure rate and worth the risk. > > Hopefully this gets us all productive again. </blockquote></div>