[fdo] 504 to gitlab.freedesktop.org

[Daniel Stone]

On Mon, 13 Jun 2022 at 05:17, Peter Hutterer <peter.hutterer at who-t.net> wrote:
> On Sun, Jun 12, 2022 at 05:57:05PM -0700, Jeremy Sequoia wrote:
> > I was going to spend a little bit of time putting out an update to XQuartz
> > to address a few bugs that I've been meaning to squash, but I'm having a bit
> > of an issue pulling down sources.
> >
> > Fetching via ssh://git@gitlab.freedesktop.org is giving me Permission denied
> > (publickey,keyboard-interactive).  I'm not sure if the latter is an infra
> > issue or if the ssh key I have stored in my gitlab account are out of date
> > (it's been about a year since I touched this).  Unfortunately, I can't seem
> > to access https://gitlab.freedesktop.org to check as it's constantly
> > presenting me a 504 Gateway timeout.
> >
> > Is anyone else able to pull sources via ssh://git@gitlab.freedesktop.org
> > right now?  Is someone looking into the 504 issue?
>
> not an fdo admin but judging by the chatter on #freedesktop: no and yes, in
> that order. seems like the infrastructure is in various stages of depositing
> fecal matter on itself and the fixes are involved enough that the admins have
> to be mentally awake, not merely physically.

Yes, that's what's happening. Our (multi-host-replicated etc) Ceph
storage setup has entered a degraded mode due to the loss of a couple
of disks - no data has been lost but the cluster is currently unhappy.
We're walking through fixing this, but have bumped into some other
issues since, including a newly-flaky network setup, and changes since
we last provisioned a new storage host.

We're working through them one by one and will have the service back
up with all our data intact - hopefully in a matter of hours but we
have no firm ETA right now.

Cheers,
Daniel