<div dir="ltr">We might want to consider pushing out the branch point a week anyway to help people get CTS in order?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 7, 2022 at 1:08 PM Ian Romanick <<a href="mailto:idr@freedesktop.org">idr@freedesktop.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Blarg. That all sounds awful. I think (hope!) I speak for everyone when <br>
I say that we all appreciate your and daniels' efforts to keep this big <br>
piece of machinery working.<br>
<br>
If the problems persist much longer, should we consider pushing out the <br>
22.0 branch point?<br>
<br>
On 1/6/22 9:36 PM, Emma Anholt wrote:<br>
> As you've probably noticed, there have been issues with git access<br>
> this week. The fd.o sysadmins are desperately trying to stay on<br>
> vacation because they do deserve a break, but have still been working<br>
> on the problem and a couple of solutions haven't worked out yet.<br>
> Hopefully we'll have some news soon.<br>
> <br>
> Due to these ongoing git timeouts, our CI runners have been getting<br>
> bogged down with stalled jobs and causing a lot of spurious failures<br>
> where the pipeline doesn't get all its jobs assigned to runners before<br>
> Marge gives up. Today, I asked daniels to bump Marge's pipeline<br>
> timeout to 4 hours (up from 1). To get MRs flowing at a similar rate<br>
> despite the longer total pipeline times, we also enabled batch mode as<br>
> described at <a href="https://github.com/smarkets/marge-bot/blob/master/README.md#batching-merge-requests" rel="noreferrer" target="_blank">https://github.com/smarkets/marge-bot/blob/master/README.md#batching-merge-requests</a>.<br>
> <br>
> It means there are now theoretical cases as described in the README<br>
> where Marge might merge a set of code that leaves main broken.<br>
> However, those cases are pretty obscure, and I expect that failure<br>
> rate to be much lower than the existing "you can merge flaky code"<br>
> failure rate and worth the risk.<br>
> <br>
> Hopefully this gets us all productive again.<br>
</blockquote></div>