Migrating Wayland & Weston to GitLab

[Daniel Stone]

Hi all,
As some of you have seen, freedesktop.org is migrating its Git hosting
to GitLab[0]. Whilst the documentation is still a little scratchy -
partly deliberate whilst we've been bootstrapping our infrastructure
and monitoring how some smaller pilot projects have gone using it -
here is what it definitely means for Wayland (wearing my fd.o admin
hat), and some of my suggestions of what we should do as a project
(wearing my 'one of many Wayland contributors' hat).

The latter suggestions are just my suggestion and opinion, which I'm
putting out there for discussion.


GitLab background
--------------------------

[fd.o admin hat on]

GitLab is a web service developed by gitlab.com, offering code
hosting, online code review, issue tracking, CI integration, hosting
of generated web pages, and so on. They offer a fully open-source
Community Edition, as well as a proprietary subscription-based
Enterprise Edition with several tiers.

fd.o currently hosts a plethora of services, including a normal git
daemon and SSH server for hosting, cgit for repository browsing,
Bugzilla for issue tracking, Mailman for mailing lists, Patchwork for
tracking patch submissions, web hosting (static pages or ikiwiki).

We also host an instance of Phabricator, which we rolled out as a test
to see if it could provide us with better workflows. Long story short,
it didn't: its issue tracking is pretty good, but code review not, as
it requires a specific 'git-phab' tool to work properly, and
supporting microcommits and rebasing workflows is an explicit
anti-goal. Its UI is also totally unsuited for the forge-like setup
fd.o has, as well as community projects: it's designed for internal
use by dedicated software teams.

All of this came up during XDC in Helsinki, where after sitting down
and talking we decided that the UI would need a lot of work to be
useful. Since then, Phabricator's development model has also closed up
with very little possibility for community contributions, which isn't
good for us. It's probably notable that the vast majority of people
who used phabricator.fd.o were from Collabora, where we use it for all
our issue tracking internally.

Last year, GNOME went through a similar search[1] for a more modern
and integrated tool which would let them have issue tracking linked
with code hosting, the ability to push branches and generate merge
requests people could review via the web from those, CI linked with
merge requests and repositories, etc. I participated in that from the
point of view of giving Phabricator a fair run out, being more
familiar with it than others. Over the course of the discussions, I
ended up supporting their conclusion that GitLab was the best tool for
them, mainly through seeing the huge strides they'd made between (at
the time) 8.x to 9.x, and subsequently.

GitLab also has a well-documented REST API, as well as external web
hooks, which can be used to extend its behaviour in ways which aren't
possible for most of our tools. Most of the functionality available
through its web interface is also available through a documented API.
Currently we deal with extensions with local changes to the codebase,
which makes upgrades painful.

One reason we're looking for a single, more modern, tool is to free up
some admin time. Not only is GitLab more featureful than our existing
suite of tools, but our biggest admin problem at the moment is exactly
those tools. Tracking the releases and deployment of all those tools
is hugely time-consuming, and all the duct tape which holds them
together is incredibly fragile. Dealing with spam is also a massive
problem across a lot of them, and that's even before you get to GDPR
issues.

Running services which don't support modern identity management is
also a problem. Users hate having to sign up in multiple places (SSH
account, Bugzilla, lists, Patchwork), where we maintain a
username/password store. None of these services support external
authentication providers (which users have complained of), none of
them support 2FA (a genuine security concern), and dealing with PII
erasure requests on the GDPR is basically unworkable.

Late last year, some of the people responsible for GNOME's migration
put us in touch with GitLab's community outreach team, who offered to
support us in our migration if that's what we wanted to do. After
going through the options, we have accepted a donation-in-kind from
them, of paying for us to run a GitLab CE instance (we declined an EE
license, or remote hosting) on Google cloud infrastructure. This will
let us replace life-expired physical machines as we wind down the
services which run on them.


Repository migration
----------------------------

[admin hat still on]

The first thing to happen is to migrate the central Git push point to
GitLab. anongit.fd.o and cgit will still work as read-only mirrors,
but you will not be able to push to git.fd.o. This would happen for
all of wayland, wayland-protocols, weston and wayland-web. libinput we
could move separately, wayland-build-tools is open to discussion (does
it still work?), and wayland-java appears to be completely abandoned.

All this would require is for everyone with push rights to set up
their GitLab account with SSH keys as per the instructions at [0]. On
a designated flag day, the push URL would become a GitLab one, and
direct pushes to git.fd.o would be rejected. Users would be able to
view and fork the repository at gitlab.fd.o, though not necessarily
able to do any more than that. Since this is easy and low-impact, I'd
propose to do this at the end of this week or early next week, if
there are no objections.


Issue tracking
-------------------

[admin hat still on]

Our plan of record is to deprecate Phabricator as rapidly as possible.
Our deployment of it is not well maintained, seems to have no future
(see above), and has so few users that it's certainly low-hanging
fruit for retiring services. Wayland's use of Phabricator is mainly
tracking and design discussion for issues filed by pq or myself. There
is also some light wiki usage, e.g. around the weston_head design.

Deprecating Bugzilla is a longer-term goal. As said, Bugzilla is
fairly hopeless at handling spam - eliminating spam takes much more
time than you might think - and is a constant source of security
worries.

We're not forcing anyone off either service at this point, but we do
provide migration of open bugs for projects using either Phabricator
or Bugzilla. The first project (GeoClue) moved its bugs from Bugzilla
to GitLab over the weekend, and we should see a few more over the rest
of the week, as well as bug migration from Phabricator.

Compared to Bugzilla, issue tracking lets projects define their own
custom milestones (releases), labels (tags), and so on, which can make
it easier to track 'what needs to be done for 1.17?'.


[admin hat off, contributor hat on]

I believe we should start triaging our Phabricator and Bugzilla issues
as soon as possible, and look to migrate them in a couple of weeks or
so, when we have a more sensible set of open issues. As said before,
having to register separately to Bugzilla is a common complaint from
users; using GitLab would also give us issue templates we could work
with.

One point we haven't (yet) sorted out is reply-by-email to deal with
GitLab issues, though anecdotally I don't think many people use that
for Wayland and Weston, so it shouldn't be a blocker.

I'm happy to do the actual migration as it needs a little bit of
manual legwork, but doing triage of currently-open issues isn't
something I can do on my own. I'm not sure how we should best
co-ordinate that - personally I do like using the 'todos' when you
have checkboxes in GitLab issues[2] to bounce things between each
other without it getting lost - but there are a lot of ways to skin
that particular cat.


CI
--

[admin hat on]

One nice thing in GitLab is CI, where just like Travis and similar,
you can create a .gitlab-ci.yml file which will cause the given
pipeline to be run on all pushes to the repository, including merge
requests. We currently support running arbitrary pipelines in Docker
containers, which we can use for build tests and so on, up to 'make
check' as well. Enabling support for running under VMs is WIP; running
things on real dedicated hardware (e.g. 'does the KMS backend still
work on Raspberry Pi') is a whole different matter though.

This doesn't require any special privilege to set up: you can fork any
repository, create a .gitlab-ci.yml test definition in your local
fork, and run all kinds of test builds, capturing the output and any
artifacts.

CI is also plumbed through to GitLab Pages. Each CI job can generate
downloadable 'artifacts', which are publicly available for a set
period of time. The way Pages works is that you run a CI pipeline that
generates a set of artifacts for Pages, which are then served from a
given domain. So wayland-web in its current form would have a CI job
which just captured all its files as artifacts, but it would be
possible to dynamically generate it from something like Jekyll, Hugo,
or whatever your generator of choice is.


[admin hat off, developer hat on]

It would be really cool to get a basic build test together for our
repositories, in various configurations. Making it easier to fix our
website would also be a massive leap forward in terms of
accessibility.


Code review
-----------------

[admin hat on]

GitLab also provides code reviews through merge requests[3], which are
like GitHub's pull requests (push to a branch or separate repo, create
a request for someone to review and merge that code), but much more
attuned to our workflows. Specifically, GitHub is geared towards
chronological commit streams (initial code, fix, fix, fix, fix, fix,
more stuff, more fixes) which ultimately get squashed into one - which
was replicated by Phabricator. GitLab does allow for the merge-ordered
microcommits we have on the list today, including allowing you to
rebase them before submitting new versions, and also not squashing the
results.

More usefully, it also allows you to create these with 'git push' then
following the URL it offers you to create a MR. Being a real git
branch allows reviewers to pull the pristine tree as the submitter had
(without lossily running through patch files), and also for CI
pipelines to be run on those branches so they can see problems without
needing the reviewer as a proxy to 'make distcheck'.

Comments on GitLab MRs can either be placed as a general comment on
the MR, or directly associated with lines of code. These comments can
either be comments (general chatter), or marked as an 'issue', which
must be explicitly marked as resolved in order to move on. This makes
it more difficult to forget review comments in subsequent revisions.


[admin hat off, contributor hat on]

Every time someone comes on to #wayland saying that they want to
submit a patch series but can't figure out how to use git-send-email,
it kills me. Every time I look at the state of Patchwork and realise
that it's a wasteland because the commit -> Patchwork hooks are
necessarily lossy, I despair a bit. That's if it even correctly
identifies and groups the patches in the first place, which it doesn't
despite two years of development specifically aimed at having it work
well for dri-devel@ and intel-gfx at .

Not having code review hooked into issue tracking means that even
after we've demanded our contributors jump through the git-send-email
hoops, their contributions can disappear into the void. I think having
it hooked up to issue tracking, with assignable labels and target
milestones, with discussions people can actually follow (rather than
Mailman archives, where any discussion spanning a month boundary is
split in two), would really lower the barrier to contributing.

We can also put a CONTRIBUTING.md file which is shown to people when
they create merge requests, outlining how we review and what we're
looking for. CI means that we can pick up the mechanical failures
early, rather than waiting for a human to come by, run the tests
themselves, and inform someone who may no longer have time or interest
at that point.

Something I've been told anecdotally by people either contributing to
or using wlc/wlroots/etc, is that one of the reasons they used that
over libweston is because our archaic setup means it's pointlessly
difficult to communicate with us. They've got a point.

My proposal is that as of the repo migration, we trial GitLab merge
requests on a couple of patchsets where we'd expect to see the need
for complex structured comments and multiple revisions. These
revisions should probably be shadow-posted to the list, to make sure
we didn't miss anything.

Once we'd been through a couple of rounds and we had a general
consensus amongst reviewers that we had a good workflow we were happy
to use, we would add that to CONTRIBUTING.md as well as README,
informing would-be contributors that we now preferred to receive
patches through GitLab, though would continue to accept patches
through the list for a short time (informing those who did submit via
the list that they should move to GitLab). Once the next release had
passed, we would stop doing that, and have GitLab MRs as our sole
point of code review.



So - thoughts?

Cheers,
Daniel

[0]: https://gitlab.freedesktop.org/freedesktop/freedesktop/wikis/home
[1]: https://wiki.gnome.org/Initiatives/DevelopmentInfrastructure
[2]: https://docs.gitlab.com/ce/workflow/todos.html
[3]: https://docs.gitlab.com/ce/user/project/merge_requests/index.html