Mesa (main): ci: Add docs for Linux Kernel uprevs

[GitLab Mirror]

Module: Mesa
Branch: main
Commit: 931495072658bc314fd494b9eb7840af0642c049
URL:    http://cgit.freedesktop.org/mesa/mesa/commit/?id=931495072658bc314fd494b9eb7840af0642c049

Author: Guilherme Gallo <guilherme.gallo at collabora.com>
Date:   Wed Jan 12 10:58:20 2022 -0300

ci: Add docs for Linux Kernel uprevs

Signed-off-by: Guilherme Gallo <guilherme.gallo at collabora.com>
Reviewed-by: Emma Anholt <emma at anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14514>

---

 docs/ci/index.rst  |  12 ++++++
 docs/ci/kernel.rst | 121 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 133 insertions(+)

diff --git a/docs/ci/index.rst b/docs/ci/index.rst
index 7f494c7538d..c6f8ef5989c 100644
--- a/docs/ci/index.rst
+++ b/docs/ci/index.rst
@@ -254,3 +254,15 @@ This section lists their documentation pages.
   :maxdepth: 1
 
   skqp
+
+
+Updating Gitlab CI Linux Kernel
+-------------------------------
+
+Gitlab CI usually runs a bleeding-edge kernel. The following documentation has
+instructions on how to uprev Linux Kernel in the Gitlab Ci ecosystem.
+
+.. toctree::
+  :maxdepth: 1
+
+  kernel
diff --git a/docs/ci/kernel.rst b/docs/ci/kernel.rst
new file mode 100644
index 00000000000..053d8ea666a
--- /dev/null
+++ b/docs/ci/kernel.rst
@@ -0,0 +1,121 @@
+Upreving Linux Kernel
+=====================
+
+Occasionally, the Gitlab CI needs a Linux Kernel update to enable new kernel
+features, device drivers, bug fixes etc to CI jobs.
+Kernel uprevs in Gitlab CI are relatively simple, but prone to lots of
+side-effects since many devices from different platforms are involved in the
+pipeline.
+
+Kernel repository
+-----------------
+
+The Linux Kernel used in the Gitlab CI is stored at the following repository:
+https://gitlab.freedesktop.org/gfx-ci/linux
+
+It is common that Mesa kernel brings some patches that were not merged on the
+Linux mainline, that is why Mesa has its own kernel version which should be used
+as the base for newer kernels.
+
+So, one should base the kernel uprev from the last tag used in the Mesa CI,
+please refer to `.gitlab-ci.yml` `KERNEL_URL` variable.
+Every tag has a standard naming: `vX.YZ-for-mesa-ci-<commit_short_SHA>`, which
+can be created via the command:
+
+:code:`git tag vX.YZ-for-mesa-ci-$(git rev-parse --short HEAD)`
+
+Building Kernel
+---------------
+
+When Mesa CI generates a new rootfs image, the Linux Kernel is built based on
+the script located at `.gitlab-ci/build-kernel.sh`.
+
+Updating Kconfigs
+^^^^^^^^^^^^^^^^^
+
+When a Kernel uprev happens, it is worth compiling and cross-compiling the
+Kernel locally, in order to update the Kconfigs accordingly.  Remember that the
+resulting Kconfig is a merge between *Mesa CI Kconfig* and *Linux tree
+defconfig* made via `merge_config.sh` script located at Linux Kernel tree.
+
+Kconfigs location
+"""""""""""""""""
+
++------------+--------------------------------------------+-------------------------------------+
+| Platform   | Mesa CI Kconfig location                   | Linux tree defconfig                |
++============+============================================+=====================================+
+| arm        | .gitlab-ci/container/arm.config            | arch/arm/configs/multi_v7_defconfig |
++------------+--------------------------------------------+-------------------------------------+
+| arm64      | .gitlab-ci/container/arm64.config          | arch/arm64/configs/defconfig        |
++------------+--------------------------------------------+-------------------------------------+
+| x86-64     | .gitlab-ci/container/x86_64.config         | arch/x86/configs/x86_64_defconfig   |
++------------+--------------------------------------------+-------------------------------------+
+
+Updating image tags
+-------------------
+
+Every kernel uprev should update 3 image tags, located at two files.
+
+:code:`.gitlab-ci.yml` tag
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+- **KERNEL_URL** for the location of the new kernel
+
+:code:`.gitlab-ci/image-tags.yml` tags
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+- **KERNEL_ROOTFS_TAG** to rebuild rootfs with the new kernel
+- **DEBIAN_X86_TEST_GL_TAG** to ensure that the new rootfs is being used by the Gitlab x86 jobs
+
+Development routine
+-------------------
+
+1. Compile the newer kernel locally for each platform.
+2. Compile device trees for ARM platforms
+3. Update Kconfigs. Are new Kconfigs necessary? Is CONFIG_XYZ_BLA deprecated? Does the `merge_config.sh` override an important config?
+4. Push a new development branch to `Kernel repository`_ based on the latest kernel tag used in Gitlab CI
+5. Hack `build-kernel.sh` script to clone kernel from your development branch
+6. Update image tags. See `Updating image tags`_
+7. Run the entire CI pipeline, all the automatic jobs should be green. If some job is red or taking too long, you will need to investigate it and probably ask for help.
+
+When the Kernel uprev is stable
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+1. Push a new tag to Mesa CI `Kernel repository`_
+2. Update KERNEL_URL `debian/x86_test-gl` job definition
+3. Open a merge request, if it is not opened yet
+
+Tips and Tricks
+---------------
+
+Compare pipelines
+^^^^^^^^^^^^^^^^^
+
+To have the most confidence that a kernel uprev does not break anything in Mesa,
+it is suggested that one runs the entire CI pipeline to check if the update affected the manual CI jobs.
+
+Step-by-step
+""""""""""""
+
+1. Create a local branch in the same git ref (should be the main branch) before branching to the kernel uprev kernel.
+2. Push this test branch
+3. Run the entire pipeline against the test branch, even the manual jobs
+4. Now do the same for the kernel uprev branch
+5. Compare the job results. If a CI job turned red on your uprev branch, it means that the kernel update broke the test. Otherwise, it should be fine.
+
+Bare-metal custom kernels
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Some CI jobs have support to plug in a custom kernel by simply changing a variable.
+This is great, since rebuilding the kernel and rootfs may takes dozens of minutes.
+
+For example, freedreno jobs `gitlab.yml` manifest support a variable named
+`BM_KERNEL`. If one puts a gz-compressed kernel URL there, the job will use that
+kernel to boot the freedreno bare-metal devices. The same works for `BM_DTB` in
+the case of device tree binaries.
+
+Careful reading of the job logs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Sometimes a job may turn to red for reasons unrelated to the kernel update, e.g.
+LAVA `tftp` timeout, problems with the freedesktop servers etc.
+So it is important to see the reason why the job turned red, and retry it if an
+infrastructure error has happened.