Mesa (master): ci/freedreno: Detect cheza HFI errors and restart the run.

[GitLab Mirror]

Module: Mesa
Branch: master
Commit: ce1bb26b0601fa0f39a471cb80d2d46329549eca
URL:    http://cgit.freedesktop.org/mesa/mesa/commit/?id=ce1bb26b0601fa0f39a471cb80d2d46329549eca

Author: Eric Anholt <eric at anholt.net>
Date:   Tue Jan 26 08:47:50 2021 -0800

ci/freedreno: Detect cheza HFI errors and restart the run.

These are intermittent (~1/day), seem to be around GPU faults (so
hopefully will go away once we clean up piglit's fault errors), and are
probably also related to our vintage firmware.  Until we can get new
hardware in the farm, just restart the flaked job.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8722>

---

 .gitlab-ci/bare-metal/cros_servo_run.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/.gitlab-ci/bare-metal/cros_servo_run.py b/.gitlab-ci/bare-metal/cros_servo_run.py
index a04dbf435a6..868f8049994 100755
--- a/.gitlab-ci/bare-metal/cros_servo_run.py
+++ b/.gitlab-ci/bare-metal/cros_servo_run.py
@@ -120,6 +120,19 @@ class CrosServoRun:
                 print("Detected cheza power management bus error, restarting run...")
                 return 2
 
+            # These HFI response errors started appearing with the introduction
+            # of piglit runs.  CosmicPenguin says:
+            #
+            # "message ID 106 isn't a thing, so likely what happened is that we
+            # got confused when parsing the HFI queue.  If it happened on only
+            # one run, then memory corruption could be a possible clue"
+            #
+            # Given that it seems to trigger randomly near a GPU fault and then
+            # break many tests after that, just restart the whole run.
+            if re.search("a6xx_hfi_send_msg.*Unexpected message id .* on the response queue", line):
+                print("Detected cheza power management bus error, restarting run...")
+                return 2
+
             result = re.search("bare-metal result: (\S*)", line)
             if result:
                 if result.group(1) == "pass":