Visibility of issues fd.o admins are faced with (Was: Re: RFC: Migration to Gitlab)

[Emil Velikov]

[Changing subject/recipients, to avoid hijacking the thread]

Hi Dan,

On Wed, 22 Aug 2018 at 17:29, Daniel Stone <daniel at fooishbar.org> wrote:
>
>  Hi,
>
> On Wed, 22 Aug 2018 at 16:02, Emil Velikov <emil.l.velikov at gmail.com> wrote:
> > On 22 August 2018 at 12:44, Daniel Vetter <daniel.vetter at ffwll.ch> wrote:
> > > I think it's time to brainstorm a bit about the gitlab migration. Basic reasons:
> > >
> > > - fd.o admins want to deprecate shell accounts and hand-rolled
> > > infrastructure, because it's a pain to keep secure&updated.
> > >
> > > - gitlab will allow us to add committers on our own, greatly
> > > simplifying that process (and offloading that task from fd.o admins).
> >
> > Random thought - I really wish the admins spoke early and louder about issues.
> > From infra to manpower and adhoc tool maintenance.
>
> I thought I mostly had it covered, but fair enough. What knowledge are
> you missing and how should it best be delivered?
>
> One first-order issue is that as documented at
> https://www.freedesktop.org/wiki/AccountRequests/ creating accounts
> requires manual admin intervention. You can also search the
> 'freedesktop.org' product on Bugzilla to see the amount of time we
> spend shuffling around GPG & SSH keys, which is about the worst
> possible use of my time. Many other people have had access to drive
> the ancient shell-script frontend to LDAP before, but for some reason
> they mostly aren't very enthusiastic about doing it all the time.
>
> In the mesa-dev@ thread about Mesa's migration, which is linked from
> my blog post, I went into quite a lot of detail about why Jenkins was
> not suitable to roll out across fd.o globally. That knowledge was
> gained from trial & error, which was a lot of time burnt. The end
> result is that we don't have any CI, except if people hang
> Travis/AppVeyor off GitHub mirrors.
>
> You've personally seen what's involved in setting up Git repository
> hooks so we can build docs. We can't give access to let people work on
> those, without giving them direct access to the literal Git repository
> itself on disk. The hooks mostly involve bespoke sets of rsync jobs
> and hand-managed SSH credentials and external services to build docs
> and so on and so forth. None of this is accessible to a newcomer who
> wants to make a non-code contribution: you have to find someone with
> access to the repo to go bash some shell scripts directly and hope you
> didn't screw it up. None of this is auditable, so if the repo
> mysteriously gets wiped, then hopefully there are some backups
> somewhere. But there definitely aren't any logs. Luckily we prevent
> most people from having access to most repositories via a mandatory
> 'shell' which only allows people to push Git; this was written by hand
> by us 12 years ago.
>
> What else? Our fork of Patchwork was until recently based on an
> ancient fork of Django, in a bespoke unreproducible production
> environment. Bugzilla is patched for spam and templates (making
> upgrades complex), yet we still have a surprising amount of spam pass
> through. There's no way to delete spam, but you have to manually move
> every bug to the 'spam' group, then go through and delete the user
> which involves copying & pasting the email and a few clicks per user.
> Mailman is patched to support Recaptcha, bringing more upgrade fun.
> ikiwiki breaks all the time because it's designed to have the
> public-access web server on the same host as the writeable Git
> repositories.
>
> Our servers are several years old and approaching life expiry, and we
> have no more spare disk. You can see in #freedesktop IRC the constant
> network connectivity issues people have to PSU almost every day. Our
> attempts to root-cause and solve those have got nowhere.
>
> I could go on, but the 'moved elsewhere' list in
> https://gitlab.freedesktop.org/freedesktop/freedesktop/issues/2
> indicates that we do have problems to solve, and that changing
> peoples' SSH keys for them isn't the best way for us to be spending
> the extremely limited time that we do have.
>
Starting with the most important - regardless of what may come as
nitpicking, I do admire the time, patience and effort that you've been
putting in fd.o.

Based on your blog-post, there are many issues beyond users usability
(think people using cgit/annongit, pw failing to parse email, etc).
And yes, I've read it a couple of times as it came out.

You mentioned many of those, so let me respin that a bit:
 - old hacky/adhoc scripts - throw those over a git repo
 - annoying and/or admin requiring workflow - throw some suggestions
about tools in ^^
 - ageing servers
 - increasing maintenance

People working on graphics [more or less familiar with some issues]
may not be the best admin/tools engineers out there.
Hence publicity, be that via blog post/XDC talk/other, is very important  IMHO.

Don't recall a blog or XDC (lighting) talk over the last 5 years - and
seemingly during that time, more issues have been pilling up.

I believe that people would have responded to such publicity, either
by writing/fixing tools, monetary or otherwise.
Perhaps not to a massive extend, but another thing to explore is
funding from X.org to hide contractors to help with some issues.
IIRC that was the case with the wiki changes a few years back.

Admittedly, I'm not familiar with the finances that X.org has.

It feels that you (and fellow admins) have been testing the limits of
your perseverance, against a mounting amount of problems and pressure.
And although you were exploring alternatives, only a limited group of
people knew the struggles you're faced with.

Thus my initial statement.
I hope you'll see what i am aiming here

It could be that I'm too naive about the impact that would have?
Yet without trying, one will never know.

HTH
Emil