[Mesa-dev] [PATCH 1/3] gitlab-ci: build Mesa using GitLab CI
Eric Anholt
eric at anholt.net
Fri Aug 10 02:35:01 UTC 2018
"Juan A. Suarez Romero" <jasuarez at igalia.com> writes:
> Creates different Docker images containing Mesa built with different
> tools (autotools, meson, scons, etc).
>
> The build is done in 3 levels: the first level creates a base image
> with all the requirements to build Mesa.
>
> The second level (based of the first one), builds different images with
> different versions of LLVM. As Gallium drivers heavily relies on LLVM,
> this will help to test the build with different LLVM versions.
>
> Finally, the latest level creates different images versions of Mesa.
> The main differences is the tool to build them: autotools, meson, scons,
> building Gallium drivers with different LLVM versions, and so on.
>
> As the purpose is just to test that everything can be built correctly,
> all the images are discarded, except one (the autotools), which is
> stored in the registry. Thus, anyone can just pull it locally and test
> against their local system.
>
> In order to build the images, Rocker is used. This is a tool that
> extends the Dockerfiles with new features that are quite interested
> here. The main features we use is the support for templating, and the
> support for mounting external directories during the image building.
> This help to use tools like ccache to improve the build speed.
>
> Signed-off-by: Juan A. Suarez Romero <jasuarez at igalia.com>
> ---
> .gitlab-ci.yml | 177 +++++++++++++++++++++++++++++++++
> gitlab-ci/Rockerfile.base | 199 ++++++++++++++++++++++++++++++++++++++
> gitlab-ci/Rockerfile.llvm | 57 +++++++++++
> gitlab-ci/Rockerfile.mesa | 145 +++++++++++++++++++++++++++
> 4 files changed, 578 insertions(+)
> create mode 100644 .gitlab-ci.yml
> create mode 100644 gitlab-ci/Rockerfile.base
> create mode 100644 gitlab-ci/Rockerfile.llvm
> create mode 100644 gitlab-ci/Rockerfile.mesa
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> new file mode 100644
> index 00000000000..5cee333dd45
> --- /dev/null
> +++ b/.gitlab-ci.yml
> @@ -0,0 +1,177 @@
> +image: docker:latest
> +
> +services:
> + - docker:dind
> +
> +stages:
> + - base
> + - llvm
> + - mesa
> +
> +variables:
> + DOCKER_IMAGE: $CI_REGISTRY_IMAGE
> + CCACHE_DIR: $CI_PROJECT_DIR/../ccache
> + LLVM: "6.0"
> +
> +cache:
> + paths:
> + - ccache/
> + key: "$CI_JOB_STAGE"
> +
> +before_script:
> + - mkdir -p ccache
> + - rm -fr ../ccache
> + - mv ccache ../
> + - export MAKEFLAGS=-j$(nproc)
> + - apk --no-cache add libc6-compat
> + - wget https://github.com/grammarly/rocker/releases/download/1.3.1/rocker-1.3.1-linux_amd64.tar.gz
> + - tar xvf rocker-1.3.1-linux_amd64.tar.gz
> + - rm rocker-1.3.1-linux_amd64.tar.gz
> + - mv rocker ..
> + - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
> +
> +after_script:
> + - mv ../ccache ./
> +
> +.build_llvm: &build_llvm
> + stage: llvm
> + cache: {}
> + script:
> + - ../rocker build -f gitlab-ci/Rockerfile.llvm --var LLVM=$LLVM
> + - docker push $CI_REGISTRY_IMAGE:llvm-$LLVM
> +
> +.build_mesa: &build_mesa
> + stage: mesa
> + script:
> + - ../rocker build -f gitlab-ci/Rockerfile.mesa --var BUILD=$BUILD --var LLVM=$LLVM --var TAG=$CI_COMMIT_REF_SLUG .
> +
> +base:
> + stage: base
> + script:
> + - DOCKERFILE_SHA256=$(cat gitlab-ci/Rockerfile.base | sha256sum | cut -c-64)
> + - IMAGE_DOCKERFILE_SHA256=$(./gitlab-ci/inspect-remote-image.sh gitlab-ci-token $CI_BUILD_TOKEN $CI_PROJECT_PATH "base" ".config.Labels[\"dockerfile.sha256\"]" || echo -n "<notfound>")
> + - if [ "$DOCKERFILE_SHA256" != "$IMAGE_DOCKERFILE_SHA256" ] ; then FORCE_BUILD=true ; fi
> + - if [ "$FORCE_BUILD" ] ; then ../rocker build -f gitlab-ci/Rockerfile.base --var DOCKERFILE_SHA256=$DOCKERFILE_SHA256 ; fi
> + - if [ "$FORCE_BUILD" ] ; then docker push $CI_REGISTRY_IMAGE:base ; fi
I think this patch file was a previous version, as patch 2 removes lines
that aren't in this block and replaces them with these ones?
> +llvm:3.3:
> + variables:
> + LLVM: "3.3"
> + <<: *build_llvm
How big do all these images end up being? Do we have any size limits on
what our CI can be uploading?
> diff --git a/gitlab-ci/Rockerfile.base b/gitlab-ci/Rockerfile.base
> new file mode 100644
> index 00000000000..a0cb5e5290d
> --- /dev/null
> +++ b/gitlab-ci/Rockerfile.base
> @@ -0,0 +1,199 @@
> +#
> +# Base image for building Mesa.
> +#
> +# ~~~
> +# rocker build -f Rockerfile.base [--attach] [--pull]
> +# ~~~
> +#
> +# Environment variables that are used in the build:
> +# - DOCKER_IMAGE: name of the final image to be tagged (default: mesa:base)
> +# - MAKEFLAGS: flags to pass to make (e.g., "-j8")
> +# - CCACHE_DIR: ccache directory (e.g, ~/.ccache)
> +#
> +
> +{{ $image := (or .Env.DOCKER_IMAGE "mesa") }}
> +
> +FROM ubuntu:xenial
> +
> +LABEL maintainer "Juan A. Suarez Romero <jasuarez at igalia.com>"
> +
> +ENV LC_ALL=C.UTF-8
> +
> +RUN apt-get update \
> + && apt-get --no-install-recommends -y install autoconf gcc g++ sudo cmake patch git \
> + automake pkg-config libtool-bin bison flex python-pip libpthread-stubs0-dev wget \
> + libxau-dev libx11-dev libxext-dev libxdamage-dev libx11-xcb-dev gettext xutils-dev \
Do we want to install libx11-xcb-dev if we're also building our own libxcb?
> + zlib1g-dev scons libelf-dev libxvmc-dev libvdpau-dev libva-dev libclc-dev ccache \
> + libpciaccess-dev libxxf86vm-dev python-setuptools python-wheel bzip2 make pciutils \
> + mesa-common-dev libexpat1-dev xz-utils libedit-dev libffi-dev libunwind-dev \
mesa-common-dev also feels funny, but I'm guessing you're planning on
building things like piglit in this image eventually.
> +RUN wget https://xorg.freedesktop.org/releases/individual/proto/glproto-1.4.14.tar.bz2 \
> + && tar -jxvf glproto-1.4.14.tar.bz2 \
> + && rm glproto-1.4.14.tar.bz2 \
> + && cd glproto-1.4.14 \
> + && ./configure \
> + && make \
> + && sudo make install \
> + && sudo ldconfig \
> + && sudo rm -fr ../glproto-1.4.14
> +
> +RUN wget https://xorg.freedesktop.org/releases/individual/proto/dri2proto-2.8.tar.bz2 \
> + && tar -jxvf dri2proto-2.8.tar.bz2 \
> + && rm dri2proto-2.8.tar.bz2 \
> + && cd dri2proto-2.8 \
> + && ./configure \
> + && make \
> + && sudo make install \
> + && sudo ldconfig \
> + && sudo rm -fr ../dri2proto-2.8
The various protocol repos are now deprecated, and the common xorgproto
has replaced them (complete with meson build system).
That doesn't have to change now, but if you end up respinning it might
be nice.
> +RUN wget https://xcb.freedesktop.org/dist/xcb-proto-1.13.tar.bz2 \
> + && tar -jxvf xcb-proto-1.13.tar.bz2 \
> + && rm xcb-proto-1.13.tar.bz2 \
> + && cd xcb-proto-1.13 \
> + && ./configure \
> + && make \
> + && sudo make install \
> + && sudo ldconfig \
> + && sudo rm -fr ../xcb-proto-1.13
> +
> +RUN wget https://xcb.freedesktop.org/dist/libxcb-1.13.tar.bz2 \
> + && tar -jxvf libxcb-1.13.tar.bz2 \
> + && rm libxcb-1.13.tar.bz2 \
> + && cd libxcb-1.13 \
> + && ./configure \
> + && make \
> + && sudo make install \
> + && sudo ldconfig \
> + && sudo rm -fr ../libxcb-1.13
> +
> +RUN wget https://xorg.freedesktop.org//releases/individual/proto/renderproto-0.11.1.tar.bz2 \
> + && tar -jxvf renderproto-0.11.1.tar.bz2 \
> + && rm renderproto-0.11.1.tar.bz2 \
> + && cd renderproto-0.11.1 \
> + && ./configure \
> + && make \
> + && sudo make install \
> + && sudo ldconfig \
> + && sudo rm -fr ../renderproto-0.11.1
> +
> +RUN wget https://xorg.freedesktop.org/releases/individual/lib/libXrender-0.9.10.tar.bz2 \
> + && tar -jxvf libXrender-0.9.10.tar.bz2 \
> + && rm libXrender-0.9.10.tar.bz2 \
> + && cd libXrender-0.9.10 \
> + && ./configure \
> + && make \
> + && sudo make install \
> + && sudo ldconfig \
> + && sudo rm -fr ../libXrender-0.9.10
Huh? What's using libXRender?
> +RUN export DRM_VERSION=`cat /home/local/mesa/configure.ac | egrep ^LIBDRM.*REQUIRED| cut -f2 -d= | sort -nr | head -n 1` \
> + && wget https://dri.freedesktop.org/libdrm/libdrm-$DRM_VERSION.tar.bz2 \
> + && tar -jxvf libdrm-$DRM_VERSION.tar.bz2 \
> + && rm libdrm-$DRM_VERSION.tar.bz2 \
> + && cd libdrm-$DRM_VERSION \
> + && ./configure --enable-freedreno --enable-vc4 --enable-etnaviv-experimental-api --enable-tegra-experimental-api \
> + && make \
> + && sudo make install \
> + && sudo ldconfig \
> + && sudo rm -fr ../libdrm-$DRM_VERSION \
> + && unset DRM_VERSION
It looks like we've bumped libdrm versions maybe 5 times this year? It
doesn't seem worth inflicting libdrm rebuild on every Mesa commit to
save the image rebuilds less than once a month.
If we don't move it to the base image, we should probably at least
switch to the meson build for it to reduce the build times.
That said, this all seems like a great start, and I'd be happy to see it
land as-is so it can be incrementally improved.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/mesa-dev/attachments/20180809/74423eb9/attachment.sig>
More information about the mesa-dev
mailing list