Mesa (main): ci: skqp: Add documentation on how to maintain skqp jobs

[GitLab Mirror]

Module: Mesa
Branch: main
Commit: 9f58275f98f8b89b5bb9fc179ed7383cd8a78c7d
URL:    http://cgit.freedesktop.org/mesa/mesa/commit/?id=9f58275f98f8b89b5bb9fc179ed7383cd8a78c7d

Author: Guilherme Gallo <guilherme.gallo at collabora.com>
Date:   Wed Jan  5 13:35:18 2022 -0300

ci: skqp: Add documentation on how to maintain skqp jobs

Signed-off-by: Guilherme Gallo <guilherme.gallo at collabora.com>
Reviewed-by: Emma Anholt <emma at anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/14146>

---

 docs/ci/index.rst |  12 +++++++
 docs/ci/skqp.rst  | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 113 insertions(+)

diff --git a/docs/ci/index.rst b/docs/ci/index.rst
index 1cb90f79b17..7f494c7538d 100644
--- a/docs/ci/index.rst
+++ b/docs/ci/index.rst
@@ -242,3 +242,15 @@ directory.  You can hack on mesa and iterate testing the build with:
 .. code-block:: console
 
     sudo docker run --rm -v `pwd`:/mesa $IMAGE ninja -C /mesa/_build
+
+
+Conformance Tests
+-----------------
+
+Some conformance tests require a special treatment to be maintained on Gitlab CI.
+This section lists their documentation pages.
+
+.. toctree::
+  :maxdepth: 1
+
+  skqp
diff --git a/docs/ci/skqp.rst b/docs/ci/skqp.rst
new file mode 100644
index 00000000000..607bc983e4d
--- /dev/null
+++ b/docs/ci/skqp.rst
@@ -0,0 +1,101 @@
+skqp
+====
+
+`skqp <https://skia.org/docs/dev/testing/skqp/>`_ stands for SKIA Quality
+Program conformance tests.  Basically, it has sets of rendering tests and unit
+tests to ensure that `SKIA <https://skia.org/>`_ is meeting its design specifications on a specific
+device.
+
+The rendering tests have support for GL, GLES and Vulkan backends and test some
+rendering scenarios.
+And the unit tests check the GPU behavior without rendering images.
+
+Tests 
+-----
+
+Render tests design
+^^^^^^^^^^^^^^^^^^^
+
+It is worth noting that `rendertests.txt` can bring some detail about each test
+expectation, so each test can have a max pixel error count, to tell skqp that it
+is OK to have at most that number of errors for that test. See also:
+https://github.com/google/skia/blob/main/tools/skqp/README_ALGORITHM.md
+
+.. _test-location:
+
+Location
+^^^^^^^^
+
+Each `rendertests.txt` and `unittest.txt` file must be located inside a specific
+subdirectory inside skqp assets directory.
+
++--------------+--------------------------------------------+
+| Test type    | Location                                   |
++==============+============================================+
+| Render tests |  `${SKQP_ASSETS_DIR}/skqp/rendertests.txt` |
++--------------+--------------------------------------------+
+| Unit tests   |  `${SKQP_ASSETS_DIR}/skqp/unittests.txt`   |
++--------------+--------------------------------------------+
+
+The `skqp-runner.sh` script will make the necessary modifications to separate
+`rendertests.txt` for each backend-driver combination. As long as the test files are located in the expected place:
+
++--------------+----------------------------------------------------------------------------------------------+
+| Test type    | Location                                                                                     |
++==============+==============================================================================================+
+| Render tests | `${MESA_REPOSITORY_DIR}/src/${GPU_DRIVER}/ci/${GPU_VERSION}-${SKQP_BACKEND}_rendertests.txt` |
++--------------+----------------------------------------------------------------------------------------------+
+| Unit tests   | `${MESA_REPOSITORY_DIR}/src/${GPU_DRIVER}/ci/${GPU_VERSION}_unittests.txt`                   |
++--------------+----------------------------------------------------------------------------------------------+
+
+Where `SKQP_BACKEND` can be:
+
+- gl: for GL backend
+- gles: for GLES backend
+- vk: for Vulkan backend
+
+Example file
+""""""""""""
+
+.. code-block:: console
+
+  src/freedreno/ci/freedreno-a630-skqp-gl_rendertests.txt
+
+- GPU_DRIVER: `freedreno`
+- GPU_VERSION: `freedreno-a630`
+- SKQP_BACKEND: `gl`
+
+.. _rendertests-design:
+
+skqp reports
+------------
+
+skqp generates reports after finishing its execution, they are located at the job
+artifacts results directory and are divided in subdirectories by rendering tests
+backends and unit
+tests. The job log has links to every generated report in order to facilitate
+the skqp debugging.
+
+Maintaining skqp on Mesa CI
+---------------------------
+
+skqp is built alongside with another binary, namely `list_gpu_unit_tests`, it is
+located in the same folder where `skqp` binary is.
+
+This binary will generate the expected `unittests.txt` for the target GPU, so
+ideally it should be executed on every skqp update and when a new device
+receives skqp CI jobs.
+
+1. Generate target unit tests for the current GPU with :code:`./list_gpu_unit_tests > unittests.txt`
+
+2. Run skqp job
+
+3. If there is a failing or crashing unit test, remove it from the corresponding `unittests.txt`
+
+4. If there is a crashing render test, remove it from the corresponding `rendertests.txt`
+
+5. If there is a failing render test, visually inspect the result from the HTML report
+    - If the render result is OK, update the max error count for that test
+    - Otherwise, or put `-1` in the same threshold, as seen in :ref:`rendertests-design`
+
+6. Remember to put the new tests files to the locations cited in :ref:`test-location`