[PATCH libinput] CI: Hook up GitLab CI
Benjamin Tissoires
benjamin.tissoires at gmail.com
Fri Jun 15 07:20:33 UTC 2018
Hi Peter,
On Fri, Jun 15, 2018 at 8:59 AM Peter Hutterer <peter.hutterer at who-t.net> wrote:
>
> From: Benjamin Tissoires <benjamin.tissoires at gmail.com>
>
> The main tasks it does is build on a few different distros as well as build
> with the various build options to make sure they work.It doesn't (yet) run the
> test suite runner because that one mostly requires device nodes to operate on.
>
> Most of the fancy is to get the docker images ready. A dnf update takes
> forever, so we don't want to do that on 10 different machines. So instead we
> build docker images with all the bits pre-installed, push that to the registry
> and use those images for testing.
>
> To speed things up, we only do that when the current image is older than a
> week. And we only do that when we push to libinput proper, so a merge request
> or pushing to your private gitlab repo will never trigger a docker image
> update - it will trigger the tests and use the docker images tough.
>
> Co-authored-by: Peter Hutterer <peter.hutterer at who-t.net>
> Signed-off-by: Benjamin Tissoires <benjamin.tissoires at gmail.com>
I have one tiny issue, I already mentioned it over IRC but posting
here for posterity.
> ---
> .gitlab-ci.yml | 411 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 411 insertions(+)
> create mode 100644 .gitlab-ci.yml
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> new file mode 100644
> index 00000000..15fe0594
> --- /dev/null
> +++ b/.gitlab-ci.yml
> @@ -0,0 +1,411 @@
> +# vim: set expandtab shiftwidth=2 tabstop=8 textwidth=0:
> +#
> +# This is a bit complicated for two reasons:
> +# - we really want to run dnf/apt/... only once, updating on the test runner for
> +# each job takes forever. So we create a docker image for each distribution
> +# tested, then run the tests on this docker image.
> +#
> +# Creating a docker image is time-consuming, so we only do so for pushes to
> +# libinput directly (not merge requests) and if the current image is 'old'.
> +#
> +# - GitLab only allows one script: set per job but we have a bunch of commands
> +# we need to re-run for each build (meson && ninja && etc). YAML cannot merge
> +# arrays templates so we're screwed.
> +#
> +# So instead we use a default_build template and override everything with
> +# variables. The only two variables that matter:
> +# MESON_PARAMS=-Denable-something=true
> +# NINJA_ARGS=dist ... to run 'ninja -C builddir dist'
> +# Note that you cannot use scripts: in any target if you expect default_build
> +# to work.
> +#
> +#
> +# All jobs must follow the naming scheme of
> +# <distribution>:<version>@activity:
> +# e.g. fedora:28 at build-default
> +
> +stages:
> + - docker_check # check if the current docker images are up to date
> + - docker_prep # rebuild the docker images if previous step failed
> + - build # for actually building things
> +
> +variables:
> + MESON_BUILDDIR: builddir
> + NINJA_ARGS: ''
> + MESON_PARAMS: ''
> + FEDORA_RPMS: 'git gcc gcc-c++ meson check-devel libudev-devel libevdev-devel doxygen graphviz valgrind binutils libwacom-devel cairo-devel gtk3-devel glib2-devel mtdev-devel'
> + UBUNTU_DEBS: 'git gcc g++ meson check libudev-dev libevdev-dev doxygen graphviz valgrind binutils libwacom-dev libcairo2-dev libgtk-3-dev libglib2.0-dev libmtdev-dev'
> + FEDORA_DOCKER_IMAGE: $CI_REGISTRY/libinput/$CI_PROJECT_NAME/fedora/$FEDORA_VERSION:latest
> + UBUNTU_DOCKER_IMAGE: $CI_REGISTRY/libinput/$CI_PROJECT_NAME/ubuntu/$UBUNTU_VERSION:latest
> + # When using docker-in-docker (dind), it's wise to use the overlayfs driver
> + # for improved performance.
> + DOCKER_DRIVER: overlay2
> +
> +.default_artifacts: &default_artifacts
> + artifacts:
> + name: "meson-logs-$CI_JOB_NAME"
> + when: always
> + expire_in: 1 week
> + paths:
> + - $MESON_BUILDDIR/meson-logs
> +
> +# The default build instructions
> +.default_build: &default_build
> + script:
> + - rm -rf $MESON_BUILDDIR
> + - meson $MESON_BUILDDIR $MESON_PARAMS
> + - meson configure $MESON_BUILDDIR
> + - ninja -C $MESON_BUILDDIR $NINJA_ARGS
> +
> +# special rule to not expose the docker creation runners to other users
> +# than those who have set up the CI to push on the registry.
> +# Users who have write access to libinput/libinput will have write
> +# access to the registry, so the libinput/libinput is a catch-all for
> +# our core developers.
> +#
> +# we can add as many users as we want by adding a new line like:
> +# - $GITLAB_USER_LOGIN == "someone"
> +.restrict_docker_creation: &restrict_docker_creation
> + only:
> + variables:
> + # Note: this is a set of logical OR, not AND
> + - $CI_PROJECT_PATH == "libinput/libinput"
> +
> +#################################################################
> +# #
> +# docker check stage #
> +# #
> +#################################################################
> +
> +.docker-check: &docker_check
> + stage: docker_check
> + services:
> + - docker:dind
> + script:
> + # get the full docker image name (CURRENT_DOCKER_IMAGE still has indirections)
> + - DOCKER_IMAGE=$(eval echo "$CURRENT_DOCKER_IMAGE")
> +
> + # get the date of the current image
> + - docker pull $DOCKER_IMAGE
> + - IMG_DATE="$(docker inspect -f '{{ .Created }}' $DOCKER_IMAGE | cut -dT -f1)"
> +
> + - TODAY_SECS=$(date -u +%s)
> + - IMG_SECS=$(date -u --date="$IMG_DATE" +%s)
> + - echo "today $TODAY_SECS, image $IMG_SECS"
> +
> + # check if image is less than a week old
> + - test $(($IMG_SECS + 604800)) -gt $TODAY_SECS
> +
> + # export an artefact telling the next stage that the image is valid
> + - touch .img_ready
> + artifacts:
> + name: image $CURRENT_DOCKER_IMAGE check
> + expire_in: 20 min
> + paths:
> + - .img_ready
> + allow_failure: true
We are using allow_failure here, which means if the job fails, the
stage will still be marked as good...
> + <<: *restrict_docker_creation
> +
> +
> +# TODO: check that the RPMS/DEBS are all in the current images
> +
> +fedora:28 at docker-check:
> + variables:
> + GIT_STRATEGY: none
> + FEDORA_VERSION: 28
> + CURRENT_DOCKER_IMAGE: $FEDORA_DOCKER_IMAGE
> + <<: *docker_check
> +
> +fedora:27 at docker-check:
> + variables:
> + GIT_STRATEGY: none
> + FEDORA_VERSION: 27
> + CURRENT_DOCKER_IMAGE: $FEDORA_DOCKER_IMAGE
> + <<: *docker_check
> +
> +ubuntu:17.10 at docker-check:
> + variables:
> + GIT_STRATEGY: none
> + UBUNTU_VERSION: "17.10"
> + CURRENT_DOCKER_IMAGE: $UBUNTU_DOCKER_IMAGE
> + <<: *docker_check
> +
> +ubuntu:18.04 at docker-check:
> + variables:
> + GIT_STRATEGY: none
> + UBUNTU_VERSION: "18.04"
> + CURRENT_DOCKER_IMAGE: $UBUNTU_DOCKER_IMAGE
> + <<: *docker_check
> +
> +
> +#################################################################
> +# #
> +# docker prep stage #
> +# #
> +#################################################################
> +
> +#
> +# This stage will recreate the docker images only if the previous
> +# stage had a build failure, i.e. the image is too old or if it is
> +# missing some dependencies.
> +#
> +.fedora at docker-prep: &fedora_docker_prep
> + stage: docker_prep
> + services:
> + - docker:dind
> + before_script:
> + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
> + script:
> + # if the check was successful, we just skip recreating the docker image
> + - test -e .img_ready && exit 0
> +
> + # create a Dockerfile with our dependencies
> + - echo "FROM fedora:$FEDORA_VERSION" > Dockerfile
> + - echo "WORKDIR /app" >> Dockerfile
> + - echo "RUN dnf upgrade -y ; dnf clean all" >> Dockerfile
> + - echo "RUN dnf install -y $FEDORA_RPMS ; dnf clean all" >> Dockerfile
> +
> + # create the docker image
> + - docker build --tag $FEDORA_DOCKER_IMAGE .
> +
> + # push the docker image to the libinput registry
> + - docker push $FEDORA_DOCKER_IMAGE
> + <<: *restrict_docker_creation
> + when: on_failure
... and this condition means we will never trigger it.
When writing this part, I also tried removing the 'allow_failure' in
the previous stage. Then the 'when' condition was all happy. But in
case the check stage was failing and we were recreating the docker
images, the next stage ('build') will not be run, because the pipeline
failed.
So one thing we could do in that case is to mark the next stage as
'always'. This would indeed force the creation of the jobs. But that
also means that when we regenerate the images, the first stage failed,
so even if the actual build stage is a success, the pipeline will be
marked as failed.
So to sum up, we should not have the 'when' here (and for ubuntu for
what matters), and do not try to micro-optimize this stage. If there
is nothing to be done, all that happens is that the container gets
started, and it just exits.
One other bike-shedding as I read through it.
> +
> +fedora:28 at docker-prep:
> + variables:
> + GIT_STRATEGY: none
> + FEDORA_VERSION: 28
> + <<: *fedora_docker_prep
> + dependencies:
> + # Note: we can not use $FEDORA_VERSION here
> + - fedora:28 at docker-check
> +
> +fedora:27 at docker-prep:
> + variables:
> + GIT_STRATEGY: none
> + FEDORA_VERSION: 27
> + <<: *fedora_docker_prep
> + dependencies:
> + # Note: we can not use $FEDORA_VERSION here
> + - fedora:27 at docker-check
> +
> +# FIXME: we should clean up the apt cache between each run
> +.ubuntu at docker-prep: &ubuntu_docker_prep
> + stage: docker_prep
> + services:
> + - docker:dind
> + before_script:
> + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
> + script:
> + # if the check was successful, we just skip recreating the docker image
> + - test -e .img_ready && exit 0
> +
> + # create a Dockerfile with our dependencies
> + - echo "FROM ubuntu:$UBUNTU_VERSION" > Dockerfile
> + - echo "WORKDIR /app" >> Dockerfile
> + - echo "RUN apt-get update" >> Dockerfile
> + - echo "RUN apt-get install -y software-properties-common" >> Dockerfile
> + - echo "RUN add-apt-repository universe" >> Dockerfile
> + - echo "RUN apt-get update" >> Dockerfile
> + - echo "RUN apt-get install -y $UBUNTU_DEBS" >> Dockerfile
> +
> + # create the docker image
> + - docker build --tag $UBUNTU_DOCKER_IMAGE .
> +
> + # push the docker image to the libinput registry
> + - docker push $UBUNTU_DOCKER_IMAGE
> + <<: *restrict_docker_creation
> + when: on_failure
> +
> +ubuntu:17.10 at docker-prep:
> + variables:
> + GIT_STRATEGY: none
> + UBUNTU_VERSION: "17.10"
> + <<: *ubuntu_docker_prep
> + dependencies:
> + # Note: we can not use $UBUNTU_VERSION here
> + - ubuntu:17.10 at docker-check
> +
> +ubuntu:18.04 at docker-prep:
> + variables:
> + GIT_STRATEGY: none
> + UBUNTU_VERSION: "18.04"
> + <<: *ubuntu_docker_prep
> + dependencies:
> + # Note: we can not use $UBUNTU_VERSION here
> + - ubuntu:18.04 at docker-check
> +
> +# Add some manual runners to be able to recreate the cache on a day
> +# the list of the rpms changed
> +
> +fedora:28 at force-docker-prep:
> + variables:
> + GIT_STRATEGY: none
> + FEDORA_VERSION: 28
> + <<: *fedora_docker_prep
> + when: manual
> + dependencies: []
> +
> +fedora:27 at force-docker-prep:
> + variables:
> + GIT_STRATEGY: none
> + FEDORA_VERSION: 27
> + <<: *fedora_docker_prep
> + when: manual
> + dependencies: []
> +
> +ubuntu:17.10 at force-docker-prep:
> + variables:
> + GIT_STRATEGY: none
> + UBUNTU_VERSION: "17.10"
> + <<: *ubuntu_docker_prep
> + when: manual
> + dependencies: []
> +
> +ubuntu:18.04 at force-docker-prep:
> + variables:
> + GIT_STRATEGY: none
> + UBUNTU_VERSION: "18.04"
> + <<: *ubuntu_docker_prep
> + when: manual
> + dependencies: []
> +
> +#################################################################
> +# #
> +# build stage #
> +# #
> +#################################################################
> +
> +#
> +# Fedora
> +#
> +
> +.fedora at template: &fedora_template
> + stage: build
> + image: $FEDORA_DOCKER_IMAGE
> + <<: *default_artifacts
> + dependencies: []
> +
> +.fedora:28 at template: &fedora_28_template
> + variables:
> + FEDORA_VERSION: 28
> + <<: *fedora_template
> +
> +.fedora:27 at template: &fedora_27_template
> + variables:
> + FEDORA_VERSION: 27
> + <<: *fedora_template
There is probably not a need for a template on this version. All that
matters is that we compile on f27, so we should remove this template
and unfold the fedora:27 at default-build below.
Cheers,
Benjamin
> +
> +fedora:28 at default-build:
> + <<: *fedora_28_template
> + <<: *default_build
> +
> +fedora:27 at default-build:
> + <<: *fedora_27_template
> + <<: *default_build
> +
> +# Below jobs are build option combinations. We only
> +# run them on one image, they shouldn't fail on one distro
> +# when they succeed on another.
> +
> +fedora:28 at build-no-libwacom:
> + <<: *fedora_28_template
> + <<: *default_build
> + variables:
> + FEDORA_VERSION: 28
> + MESON_PARAMS: "-Dlibwacom=false"
> +
> +fedora:28 at build-no-libwacom-nodeps:
> + <<: *fedora_28_template
> + <<: *default_build
> + variables:
> + FEDORA_VERSION: 28
> + MESON_PARAMS: "-Dlibwacom=false"
> + before_script:
> + - dnf remove -y libwacom libwacom-devel
> +
> +fedora:28 at build-no-docs:
> + <<: *fedora_28_template
> + <<: *default_build
> + variables:
> + FEDORA_VERSION: 28
> + MESON_PARAMS: "-Ddocumentation=false"
> +
> +fedora:28 at build-no-docs-nodeps:
> + <<: *fedora_28_template
> + <<: *default_build
> + variables:
> + FEDORA_VERSION: 28
> + MESON_PARAMS: "-Ddocumentation=false"
> + before_script:
> + - dnf remove -y doxygen graphviz
> +
> +fedora:28 at build-no-debuggui:
> + <<: *fedora_28_template
> + <<: *default_build
> + variables:
> + FEDORA_VERSION: 28
> + MESON_PARAMS: "-Ddebug-gui=false"
> +
> +fedora:28 at build-no-debuggui-nodeps:
> + <<: *fedora_28_template
> + <<: *default_build
> + variables:
> + FEDORA_VERSION: 28
> + MESON_PARAMS: "-Ddebug-gui=false"
> + before_script:
> + - dnf remove -y gtk3-devel
> +
> +fedora:28 at build-no-tests:
> + <<: *fedora_28_template
> + <<: *default_build
> + variables:
> + FEDORA_VERSION: 28
> + MESON_PARAMS: "-Dtests=false"
> +
> +fedora:28 at build-no-tests-nodeps:
> + <<: *fedora_28_template
> + <<: *default_build
> + variables:
> + FEDORA_VERSION: 28
> + MESON_PARAMS: "-Dtests=false"
> + before_script:
> + - dnf remove -y check-devel
> +
> +fedora:28 at scan-build:
> + <<: *fedora_28_template
> + <<: *default_build
> + variables:
> + FEDORA_VERSION: 28
> + NINJA_ARGS: scan-build
> + before_script:
> + - dnf install -y clang-analyzer findutils
> + after_script:
> + - test ! -d $MESON_BUILDDIR/meson-logs/scanbuild && exit 0
> + - test $(find $MESON_BUILDDIR/meson-logs/scanbuild -maxdepth 0 ! -empty -exec echo "not empty" \; | wc -l) -eq 0 && exit 0
> + - echo "Check scan-build results"
> + - /bin/false
> +
> +#
> +# Ubuntu
> +#
> +
> +.ubuntu at template: &ubuntu_template
> + stage: build
> + image: $UBUNTU_DOCKER_IMAGE
> + <<: *default_artifacts
> + dependencies: []
> +
> +ubuntu:17.10 at default-build:
> + variables:
> + UBUNTU_VERSION: "17.10"
> + <<: *ubuntu_template
> + <<: *default_build
> +
> +ubuntu:18.04 at default-build:
> + variables:
> + UBUNTU_VERSION: "17.10"
> + <<: *ubuntu_template
> + <<: *default_build
> --
> 2.14.4
>
More information about the wayland-devel
mailing list