[Mesa-dev] [PATCH 0/3] Use GitLab CI to build Mesa

[Daniel Stone]

Hi Juan,

On Wed, 8 Aug 2018 at 16:45, Juan A. Suarez Romero <jasuarez at igalia.com> wrote:
> This is a first part of a more complete proposal to use GitLab CI to build and
> test Mesa. This first part just adds the required pieces to build Mesa, using the
> different supported tools (meson, autotools, and scons).

This is great - I'm super excited to see it happen!

> Unfortunately, Rocker is a tool that is not maintained anymore
> (https://github.com/grammarly/rocker). We still use it because it provides
> everything we need, and fortunately we don't need any more features.
>
> Maybe there are other alternatives out there, but we were happy with this and
> hence our proposal. If we want to use Docker rather than Rocker, then we could
> use template tools like Jinja, and forget about caching during build, which will
> impact on the build time.

This bit is a bit concerning, especially since it makes CI harder to
approach for people who might want to work on it. There are quite a
few alternate tools (buildah comes to mind, as well as umoci/skopeo)
which might end up being better, particularly for scriptability. But
I'm not volunteering to do that work, so take this with a grain of
salt!

> ## Involved stages
>
> The dependencies required to build Mesa doesn't change very frequently, so
> building them everytime is a waste of time. As Docker allows to create images
> based on the content of other images, we have defined the setup in several
> stages.
>
> On a first stage a "base" image is built. This image contains almost all the
> dependencies required to build Mesa. Worth to mention that libdrm is excluded
> here, as this is a dependency that really changes quite frequently, so we
> postpone the installation for further stages.
>
> One we have the "base" image, we create different images with the different LLVM
> compilers. This ensure that when using a specific image we only have that LLVM
> version, and not any other.
>
> An important point here is that, these builts appears in the pipeline, they are
> not actually built if not required. That is, in the case of the base image, if
> the Rockerfile used to create the image has changed with respect to the one used
> to create the image that is already in the registry, then the image is rebuilt
> (as this means something changed, very likely some dependency). But if the
> Rockerfile didn't change, then there is no need to rebuild the image, and just
> keep using the one already in the registry. This is also done for the LLVM
> images. This helps to improve the speed, as most of the times they don't need to
> be built again.

It would be nice to have a GitLab CI variable which can be used to
force rebuilds of these regardless.

> The third stage is the one that builds Mesa itself. Here we just define which
> tool to use and which LLVM version. This is done by passing the right parameters
> to the `rocker build` tool. It will pick the right base image, install the
> missing dependencies (mainly, libdrm), select which drivers should be built
> (based on the LLVM version and parsing the configure.ac file), and create the
> image.

You can eke out a little bit of a speed improvement by using the
go-faster runes from before 'apt-get update' here:
    https://gitlab.freedesktop.org/wayland/weston/blob/master/.gitlab-ci.yml#L6

Anyway, this all looks great, and I'm really excited to see it! The series is:
Acked-by: Daniel Stone <daniels at collabora.com>

... but please do not deploy it just yet. Right now we only have a
single CI runner for all of fd.o, which is a pretty elderly (SNB?)
Xeon, on a different continent from our GitLab instance. The network
is chokingly bad, but even if it weren't we'd be bottlenecked on
machine time. I'm working around to try to find sponsorship to get us
some more runners, located in decent data centres in the US, and will
let you know as soon as anything happens there. In the meantime, if
you have any ideas or know any people who would be interested in
sponsoring runners (very ideally US-based) or just covering the bill
for time on cloud VMs, please let me know!

Cheers,
Daniel