Mesa (master): ci: Add scripts for controlling bare-metal chezas.
GitLab Mirror
gitlab-mirror at kemper.freedesktop.org
Fri May 29 18:24:13 UTC 2020
Module: Mesa
Branch: master
Commit: c89a749f66527caf72c43b433dc27de1594a87f6
URL: http://cgit.freedesktop.org/mesa/mesa/commit/?id=c89a749f66527caf72c43b433dc27de1594a87f6
Author: Eric Anholt <eric at anholt.net>
Date: Fri May 15 09:57:25 2020 -0700
ci: Add scripts for controlling bare-metal chezas.
This will let us:
- deploy kernels for testing code depending on new kernel featuers
- Ensure a pristine state in the HW before starting our tests
- Avoid disk rot on the chezas taking them out (we'd lost 3/9 in a few
months).
Reviewed-by: Kristian H. Kristensen <hoegsberg at google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5247>
---
.gitlab-ci/bare-metal/README.md | 88 +++++++++++++++++++++++--------
.gitlab-ci/bare-metal/cros-servo.sh | 97 +++++++++++++++++++++++++++++++++++
.gitlab-ci/bare-metal/fastboot.sh | 54 ++-----------------
.gitlab-ci/bare-metal/rootfs-setup.sh | 55 ++++++++++++++++++++
.gitlab-ci/bare-metal/write-serial.py | 11 ++++
5 files changed, 232 insertions(+), 73 deletions(-)
diff --git a/.gitlab-ci/bare-metal/README.md b/.gitlab-ci/bare-metal/README.md
index 3bbfab11cfb..843d168c455 100644
--- a/.gitlab-ci/bare-metal/README.md
+++ b/.gitlab-ci/bare-metal/README.md
@@ -3,40 +3,76 @@
Testing Mesa with gitlab-runner running on the devices being tested
(DUTs) proved to be too unstable, so this set of scripts is for
running Mesa testing on bare-metal boards connected to a separate
-system using gitlab-runner. Currently only "fastboot" devices are
-supported.
+system using gitlab-runner. Currently only "fastboot" and "ChromeOS
+Servo" devices are supported.
In comparison with LAVA, this doesn't involve maintaining a separate
webservice with its own job scheduler and replicating jobs between the
two. It also places more of the board support in git, instead of
-webservice configuration. Most importantly, it doesn't download the
-rootfs as artifacts on each job, so we can avoid traffic to
-freedesktop.org. On the other hand, the serial interactions and
-bootloader support are more primitive.
+webservice configuration. On the other hand, the serial interactions
+and bootloader support are more primitive.
-## Requirements
+## Requirements (fastboot)
This testing requires power control of the DUTs by the gitlab-runner
machine, since this is what we use to reset the system and get back to
a pristine state at the start of testing.
-We require access to the console output from the gitlb-runner system,
-since that is how we get the final results back from te tests. You
+We require access to the console output from the gitlab-runner system,
+since that is how we get the final results back from the tests. You
should probably have the console on a serial connection, so that you
can see bootloader progress.
The boards need to be able to have a kernel/initramfs supplied by the
gitlab-runner system, since the initramfs is what contains the Mesa
-testing payload. Currently only "fastboot" devices are supported.
+testing payload.
The boards should have networking, so that (in a future iteration of
this code) we can extract the dEQP .xml results to artifacts on
gitlab.
+## Requirements (servo)
+
+For servo-connected boards, we can use the EC connection for power
+control to reboot the board. However, loading a kernel is not as easy
+as fastboot, so we assume your bootloader can do TFTP, and that your
+gitlab-runner mounts the runner's tftp directory specific to the board
+at /tftp in the container.
+
+Since we're going the TFTP route, we also use NFS root. This avoids
+packing the rootfs and sending it to the board as a ramdisk, which
+means we can support larger rootfses (for piglit or tracie testing),
+at the cost of needing more storage on the runner.
+
+Telling the board about where its TFTP and NFS should come from is
+done using dnsmasq on the runner host. For example, this snippet in
+the dnsmasq.conf.d in the google farm, with the gitlab-runner host we
+call "servo".
+
+```
+dhcp-host=1c:69:7a:0d:a3:d3,10.42.0.10,set:servo
+
+# Fixed dhcp addresses for my sanity, and setting a tag for
+# specializing other DHCP options
+dhcp-host=a0:ce:c8:c8:d9:5d,10.42.0.11,set:cheza1
+dhcp-host=a0:ce:c8:c8:d8:81,10.42.0.12,set:cheza2
+
+# Specify the next server, watch out for the double ',,'. The
+# filename didn't seem to get picked up by the bootloader, so we use
+# tftp-unique-root and mount directories like
+# /srv/tftp/10.42.0.11/jwerner/cheza as /tftp in the job containers.
+tftp-unique-root
+dhcp-boot=tag:cheza1,cheza1/vmlinuz,,10.42.0.10
+dhcp-boot=tag:cheza2,cheza2/vmlinuz,,10.42.0.10
+
+dhcp-option=tag:cheza1,option:root-path,/srv/nfs/cheza1
+dhcp-option=tag:cheza2,option:root-path,/srv/nfs/cheza2
+```
+
## Setup
Each board will be registered in fd.o gitlab. You'll want something
-like this to register:
+like this to register a fastboot board:
```
sudo gitlab-runner register \
@@ -52,27 +88,35 @@ sudo gitlab-runner register \
--non-interactive
```
+For a servo board, you'll need to also volume mount the board's NFS
+root dir at /nfs and TFTP kernel directory at /tftp.
+
The registration token has to come from a fd.o gitlab admin going to
https://gitlab.freedesktop.org/admin/runners
-The name scheme for Google's lab is google-freedreno-boardname-nn, and
-our tag is google-freedreno-db410c. The tag is what identifies a
-board type so that board-specific jobs can be dispatched into that
-pool.
+The name scheme for Google's lab is google-freedreno-boardname-n, and
+our tag is something like google-freedreno-db410c. The tag is what
+identifies a board type so that board-specific jobs can be dispatched
+into that pool.
We need privileged mode and the /dev bind mount in order to get at the
serial console and fastboot USB devices (--device arguments don't
apply to devices that show up after container start, which is the case
-with fastboot). We use host network mode so that we can (in the
-future) spin up a server to collect XML results.
-
-Once you've added your boards, you're going to need to specify the
-board-specific env vars, adding something like this `environment` line
-to each runner in `/etc/gitlab-runner/config.toml`
+with fastboot, and the servo serial devices are acctually links to
+/dev/pts). We use host network mode so that we can (in the future)
+spin up a server to collect XML results for fastboot.
+
+Once you've added your boards, you're going to need to add a little
+more customization in `/etc/gitlab-runner/config.toml`. First, add
+`concurrent = <number of boards>` at the top ("we should have up to
+this many jobs running managed by this gitlab-runner"). Then for each
+board's runner, set `limit = 1` ("only 1 job served by this board at a
+time"). Finally, add the board-specific environment variables
+required by your bare-metal script, something like:
```
[[runners]]
- name = "google-freedreno-db410c-01"
+ name = "google-freedreno-db410c-1"
environment = ["BM_SERIAL=/dev/ttyDB410c8", "BM_POWERUP=google-power-up.sh 8", "BM_FASTBOOT_SERIAL=15e9e390"]
```
diff --git a/.gitlab-ci/bare-metal/cros-servo.sh b/.gitlab-ci/bare-metal/cros-servo.sh
new file mode 100755
index 00000000000..b67c1f2ad65
--- /dev/null
+++ b/.gitlab-ci/bare-metal/cros-servo.sh
@@ -0,0 +1,97 @@
+#!/bin/bash
+
+# Boot script for Chrome OS devices attached to a servo debug connector, using
+# NFS and TFTP to boot.
+
+# We're run from the root of the repo, make a helper var for our paths
+BM=$CI_PROJECT_DIR/.gitlab-ci/bare-metal
+
+# Runner config checks
+if [ -z "$BM_SERIAL" ]; then
+ echo "Must set BM_SERIAL in your gitlab-runner config.toml [[runners]] environment"
+ echo "This is the CPU serial device."
+ exit 1
+fi
+
+if [ -z "$BM_SERIAL_EC" ]; then
+ echo "Must set BM_SERIAL in your gitlab-runner config.toml [[runners]] environment"
+ echo "This is the EC serial device for controlling board power"
+ exit 1
+fi
+
+if [ ! -d /nfs ]; then
+ echo "NFS rootfs directory needs to be mounted at /nfs by the gitlab runner"
+ exit 1
+fi
+
+if [ ! -d /tftp ]; then
+ echo "TFTP directory for this board needs to be mounted at /tftp by the gitlab runner"
+ exit 1
+fi
+
+# job config checks
+if [ -z "$BM_KERNEL" ]; then
+ echo "Must set BM_KERNEL to your board's kernel FIT image"
+ exit 1
+fi
+
+if [ -z "$BM_ROOTFS" ]; then
+ echo "Must set BM_ROOTFS to your board's rootfs directory in the job's variables"
+ exit 1
+fi
+
+if [ -z "$BM_CMDLINE" ]; then
+ echo "Must set BM_CMDLINE to your board's kernel command line arguments"
+ exit 1
+fi
+
+set -ex
+
+# Create the rootfs in the NFS directory. rm to make sure it's in a pristine
+# state, since it's volume-mounted on the host.
+rm -rf /nfs/*
+mkdir -p /nfs/results
+. $BM/rootfs-setup.sh /nfs
+
+# Set up the TFTP kernel/cmdline. When we support more than one board with
+# this method, we'll need to do some check on the runner name or something.
+rm -rf /tftp/*
+cp $BM_KERNEL /tftp/vmlinuz
+echo "$BM_CMDLINE" > /tftp/cmdline
+
+# Start watching serials, and power up the device.
+$BM/serial-buffer.py $BM_SERIAL_EC | tee serial-ec-output.txt | sed -u 's|^|SERIAL-EC> |g' &
+$BM/serial-buffer.py $BM_SERIAL | tee serial-output.txt | sed -u 's|^|SERIAL-CPU> |g' &
+while [ ! -e serial-output.txt ]; do
+ sleep 1
+done
+# Flush any partial commands in the EC's prompt, then ask for a reboot.
+$BM/write-serial.py $BM_SERIAL_EC ""
+$BM/write-serial.py $BM_SERIAL_EC reboot
+
+# This is emitted right when the bootloader pauses to check for input. Emit a
+# ^N character to request network boot, because we don't have a
+# direct-to-netboot firmware on cheza.
+$BM/expect-output.sh serial-output.txt "load_archive: loading locale_en.bin"
+$BM/write-serial.py $BM_SERIAL `printf '\016'`
+
+# Wait for the device to complete the deqp run
+$BM/expect-output.sh serial-output.txt "DEQP RESULT"
+
+# power down the CPU on the device
+$BM/write-serial.py $BM_SERIAL_EC 'power off'
+
+set -ex
+
+# Bring artifacts back from the NFS dir to the build dir where gitlab-runner
+# will look for them. Note that results/ may already exist, so be careful
+# with cp.
+mkdir -p results
+cp -Rp /nfs/results/. results/
+
+set +e
+if grep -q "DEQP RESULT: pass" serial-output.txt; then
+ exit 0
+else
+ exit 1
+fi
diff --git a/.gitlab-ci/bare-metal/fastboot.sh b/.gitlab-ci/bare-metal/fastboot.sh
index 221d8efc26e..124309e8abe 100755
--- a/.gitlab-ci/bare-metal/fastboot.sh
+++ b/.gitlab-ci/bare-metal/fastboot.sh
@@ -44,57 +44,9 @@ fi
set -ex
-# Copy the rootfs to a temporary for our setup, as I believe changes to the
-# container can end up impacting future runs.
-cp -Rp $BM_ROOTFS rootfs
-
-# Set up the init script that brings up the system.
-cp $BM/init.sh rootfs/init
-
-set +x
-# Pass through relevant env vars from the gitlab job to the baremetal init script
-touch rootfs/set-job-env-vars.sh
-chmod +x rootfs/set-job-env-vars.sh
-for var in \
- CI_COMMIT_BRANCH \
- CI_COMMIT_TITLE \
- CI_JOB_ID \
- CI_JOB_URL \
- CI_MERGE_REQUEST_SOURCE_BRANCH_NAME \
- CI_MERGE_REQUEST_TITLE \
- CI_NODE_INDEX \
- CI_NODE_TOTAL \
- CI_PIPELINE_ID \
- CI_RUNNER_DESCRIPTION \
- DEQP_CASELIST_FILTER \
- DEQP_EXPECTED_RENDERER \
- DEQP_NO_SAVE_RESULTS \
- DEQP_PARALLEL \
- DEQP_RUN_SUFFIX \
- DEQP_VER \
- FD_MESA_DEBUG \
- FLAKES_CHANNEL \
- IR3_SHADER_DEBUG \
- NIR_VALIDATE \
- ; do
- val=`echo ${!var} | sed 's|"||g'`
- echo "export $var=\"${val}\"" >> rootfs/set-job-env-vars.sh
-done
-echo "Variables passed through:"
-cat rootfs/set-job-env-vars.sh
-set -x
-
-# Add the Mesa drivers we built, and make a consistent symlink to them.
-mkdir -p rootfs/$CI_PROJECT_DIR
-tar -C rootfs/$CI_PROJECT_DIR/ -xf $CI_PROJECT_DIR/artifacts/install.tar
-ln -sf $CI_PROJECT_DIR/install rootfs/install
-
-# Copy the deqp runner script and metadata.
-cp .gitlab-ci/deqp-runner.sh rootfs/deqp/.
-cp .gitlab-ci/$DEQP_SKIPS rootfs/$CI_PROJECT_DIR/install/deqp-skips.txt
-if [ -n "$DEQP_EXPECTED_FAILS" ]; then
- cp .gitlab-ci/$DEQP_EXPECTED_FAILS rootfs/$CI_PROJECT_DIR/install/deqp-expected-fails.txt
-fi
+# Create the rootfs in a temp dir
+mkdir rootfs
+. .gitlab-ci/bare-metal/rootfs-setup.sh rootfs
# Finally, pack it up into a cpio rootfs.
pushd rootfs
diff --git a/.gitlab-ci/bare-metal/rootfs-setup.sh b/.gitlab-ci/bare-metal/rootfs-setup.sh
new file mode 100644
index 00000000000..f8c9fa42ba0
--- /dev/null
+++ b/.gitlab-ci/bare-metal/rootfs-setup.sh
@@ -0,0 +1,55 @@
+#!/bin/bash
+
+rootfs_dst=$1
+
+# Copy the rootfs to a temporary for our setup, as I believe changes to the
+# container can end up impacting future runs.
+cp -Rp $BM_ROOTFS/. $rootfs_dst
+
+# Set up the init script that brings up the system.
+cp $BM/init.sh $rootfs_dst/init
+
+set +x
+# Pass through relevant env vars from the gitlab job to the baremetal init script
+touch $rootfs_dst/set-job-env-vars.sh
+chmod +x $rootfs_dst/set-job-env-vars.sh
+for var in \
+ CI_COMMIT_BRANCH \
+ CI_COMMIT_TITLE \
+ CI_JOB_ID \
+ CI_JOB_URL \
+ CI_MERGE_REQUEST_SOURCE_BRANCH_NAME \
+ CI_MERGE_REQUEST_TITLE \
+ CI_NODE_INDEX \
+ CI_NODE_TOTAL \
+ CI_PIPELINE_ID \
+ CI_RUNNER_DESCRIPTION \
+ DEQP_CASELIST_FILTER \
+ DEQP_EXPECTED_RENDERER \
+ DEQP_NO_SAVE_RESULTS \
+ DEQP_PARALLEL \
+ DEQP_RUN_SUFFIX \
+ DEQP_VER \
+ FD_MESA_DEBUG \
+ FLAKES_CHANNEL \
+ IR3_SHADER_DEBUG \
+ NIR_VALIDATE \
+ ; do
+ val=`echo ${!var} | sed 's|"||g'`
+ echo "export $var=\"${val}\"" >> $rootfs_dst/set-job-env-vars.sh
+done
+echo "Variables passed through:"
+cat $rootfs_dst/set-job-env-vars.sh
+set -x
+
+# Add the Mesa drivers we built, and make a consistent symlink to them.
+mkdir -p $rootfs_dst/$CI_PROJECT_DIR
+tar -C $rootfs_dst/$CI_PROJECT_DIR/ -xf $CI_PROJECT_DIR/artifacts/install.tar
+ln -sf $CI_PROJECT_DIR/install $rootfs_dst/install
+
+# Copy the deqp runner script and metadata.
+cp .gitlab-ci/deqp-runner.sh $rootfs_dst/deqp/
+cp .gitlab-ci/$DEQP_SKIPS $rootfs_dst/$CI_PROJECT_DIR/install/deqp-skips.txt
+if [ -n "$DEQP_EXPECTED_FAILS" ]; then
+ cp .gitlab-ci/$DEQP_EXPECTED_FAILS $rootfs_dst/$CI_PROJECT_DIR/install/deqp-expected-fails.txt
+fi
diff --git a/.gitlab-ci/bare-metal/write-serial.py b/.gitlab-ci/bare-metal/write-serial.py
new file mode 100755
index 00000000000..6a57ea754af
--- /dev/null
+++ b/.gitlab-ci/bare-metal/write-serial.py
@@ -0,0 +1,11 @@
+#!/usr/bin/python3
+
+import sys
+import serial
+
+dev = sys.argv[1]
+command = sys.argv[2] + '\n'
+
+ser = serial.Serial(dev, 115200, timeout=5)
+ser.write(command.encode())
+ser.close()
More information about the mesa-commit
mailing list