[Intel-gfx] [IMPORTANT] Intel GFX CI affected by number of issues in past days

[Musial, Ewelina]

Hello,

In past few days Public GFX CI was affected by two major issues:

# Not all BAT machines were able to reboot successfully after several recent kernel deadlocks. Each such scenario required manual intervention to get machines back into execution. WIP to rootcause why hard reboot is not properly triggered and to fix the problem.
Issues started ~CI_DRM_12790
Idle runs were disabled till we will not have fix applied.

# Reporting shards was blocked over weekend. Execution was happening but we were not able to report results and queue was growing. We found two areas for improvement on which we are working now.
Reporting was unblocked yesterday morning and now queue is empty. We are monitoring situation.

We are very sorry for inconveniences.

Additionally, we would like to apologize for lack of communication from CI team. We are working to improve it and provide you better support in future.
As of now, please contact us on IRC #intel-gfx-ci and via email.

Regards,
Ewelina
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/intel-gfx/attachments/20230307/b2824b7b/attachment.htm>