[Mesa-dev] [PATCH 0/3] Use GitLab CI to build Mesa

[Juan A. Suarez Romero]

On Wed, 2018-08-08 at 17:40 +0100, Daniel Stone wrote:
> Hi Juan,
> 
> On Wed, 8 Aug 2018 at 16:45, Juan A. Suarez Romero <jasuarez at igalia.com> wrote:
> > This is a first part of a more complete proposal to use GitLab CI to build and
> > test Mesa. This first part just adds the required pieces to build Mesa, using the
> > different supported tools (meson, autotools, and scons).
> 
> This is great - I'm super excited to see it happen!
> 
> > Unfortunately, Rocker is a tool that is not maintained anymore
> > (https://github.com/grammarly/rocker). We still use it because it provides
> > everything we need, and fortunately we don't need any more features.
> > 
> > Maybe there are other alternatives out there, but we were happy with this and
> > hence our proposal. If we want to use Docker rather than Rocker, then we could
> > use template tools like Jinja, and forget about caching during build, which will
> > impact on the build time.
> 
> This bit is a bit concerning, especially since it makes CI harder to
> approach for people who might want to work on it. There are quite a
> few alternate tools (buildah comes to mind, as well as umoci/skopeo)
> which might end up being better, particularly for scriptability. But
> I'm not volunteering to do that work, so take this with a grain of
> salt!
> 

Yes, I knew whis could be a concerning point. There are two points to cover:

- Dockerfile templates, to allow complex building from single Dockerfile
- Allow external mount on Docker build

buildah indeed provides a way for the later. Still, we need to fix the the
former, unless we drop the Dockerfile and use whatever config syntax uses
buildah. Otherwise, something like a Jinja template or similar, and proper
tools, could fix the problem.

One of the things we decided to continue, and keep using Rocker, is that this
tool fixes both problems at the same time. And the syntax of the Rockerfiles is
the same as the Dockerfiles, but with more options. So anyone familiarized with
Docker will have no problems with the Rockerfiles.

A final reason for Rocker is how easy is to install: it is a single executable
file. This means that if someone needs to build the containers locally, they
only need to download a single file and build the containers, no matter the
distro they are using.


Anyway, I'm not paid by Rocker :). So I'm fine if we need to search for an
alternative.



> > ## Involved stages
> > 
> > The dependencies required to build Mesa doesn't change very frequently, so
> > building them everytime is a waste of time. As Docker allows to create images
> > based on the content of other images, we have defined the setup in several
> > stages.
> > 
> > On a first stage a "base" image is built. This image contains almost all the
> > dependencies required to build Mesa. Worth to mention that libdrm is excluded
> > here, as this is a dependency that really changes quite frequently, so we
> > postpone the installation for further stages.
> > 
> > One we have the "base" image, we create different images with the different LLVM
> > compilers. This ensure that when using a specific image we only have that LLVM
> > version, and not any other.
> > 
> > An important point here is that, these builts appears in the pipeline, they are
> > not actually built if not required. That is, in the case of the base image, if
> > the Rockerfile used to create the image has changed with respect to the one used
> > to create the image that is already in the registry, then the image is rebuilt
> > (as this means something changed, very likely some dependency). But if the
> > Rockerfile didn't change, then there is no need to rebuild the image, and just
> > keep using the one already in the registry. This is also done for the LLVM
> > images. This helps to improve the speed, as most of the times they don't need to
> > be built again.
> 
> It would be nice to have a GitLab CI variable which can be used to
> force rebuilds of these regardless.
> 

Oh, right! In our system, what we do is to use the scheduled pipelines to force
this rebuild. The schedule is run once per week. I'll include this option then.


> > The third stage is the one that builds Mesa itself. Here we just define which
> > tool to use and which LLVM version. This is done by passing the right parameters
> > to the `rocker build` tool. It will pick the right base image, install the
> > missing dependencies (mainly, libdrm), select which drivers should be built
> > (based on the LLVM version and parsing the configure.ac file), and create the
> > image.
> 
> You can eke out a little bit of a speed improvement by using the
> go-faster runes from before 'apt-get update' here:
>     https://gitlab.freedesktop.org/wayland/weston/blob/master/.gitlab-ci.yml#L6
> 

Cool! I think we can include it in the base image, so it gets propagated to rest
of the builds.

> Anyway, this all looks great, and I'm really excited to see it! The series is:
> Acked-by: Daniel Stone <daniels at collabora.com>
> 
> ... but please do not deploy it just yet. Right now we only have a
> single CI runner for all of fd.o, which is a pretty elderly (SNB?)
> Xeon, on a different continent from our GitLab instance. The network
> is chokingly bad, but even if it weren't we'd be bottlenecked on
> machine time. I'm working around to try to find sponsorship to get us
> some more runners, located in decent data centres in the US, and will
> let you know as soon as anything happens there. In the meantime, if
> you have any ideas or know any people who would be interested in
> sponsoring runners (very ideally US-based) or just covering the bill
> for time on cloud VMs, please let me know!
> 

Sure. Probably there are other improvements I can do meanwhile. I'll keep
unpushed until everything ready and you grant green light.

> Cheers,
> Daniel
>