[PATCH 2/2] drm/xe/guc: Port over the slow GuC loading support from i915
John Harrison
john.c.harrison at intel.com
Thu Apr 4 19:06:17 UTC 2024
On 4/4/2024 11:51, Lucas De Marchi wrote:
> I was checking the new version for this patch and remembered I had
> already said somthing. You replied and I forgot to reply back. See
> below.
>
> Other feedback I will give in the new version (if there is any).
>
> On Tue, Feb 06, 2024 at 05:51:24PM -0800, John Harrison wrote:
>> On 2/6/2024 13:36, Lucas De Marchi wrote:
>>> On Tue, Feb 06, 2024 at 12:11:51PM -0800, John.C.Harrison at Intel.com
>>> wrote:
>>>> From: John Harrison <John.C.Harrison at Intel.com>
>>>>
>>>> GuC loading can take longer than it is supposed to for various
>>>> reasons. So add in the code to cope with that and to report it when it
>>>> happens. There are also many different reasons why GuC loading can
>>>> fail, so add in the code for checking for those and for reporting
>>>> issues in a meaningful manner rather than just hitting a timeout and
>>>> saying 'fail: status = %x'.
>>>>
>>>> Also, remove the 'FIXME' comment about an i915 bug that has never been
>>>> applicable to Xe!
>>>>
>>>> Signed-off-by: John Harrison <John.C.Harrison at Intel.com>
>>>> ---
>>>> drivers/gpu/drm/xe/abi/guc_errors_abi.h | 26 +++-
>>>> drivers/gpu/drm/xe/regs/xe_guc_regs.h | 2 +
>>>> drivers/gpu/drm/xe/xe_guc.c | 197 +++++++++++++++++++-----
>>>> drivers/gpu/drm/xe/xe_macros.h | 32 ++++
>>>> 4 files changed, 214 insertions(+), 43 deletions(-)
>>>>
>>>> diff --git a/drivers/gpu/drm/xe/abi/guc_errors_abi.h
>>>> b/drivers/gpu/drm/xe/abi/guc_errors_abi.h
>>>> index ec83551bf9c0..d0b5fed6876f 100644
>>>> --- a/drivers/gpu/drm/xe/abi/guc_errors_abi.h
>>>> +++ b/drivers/gpu/drm/xe/abi/guc_errors_abi.h
>>>> @@ -7,8 +7,12 @@
>>>> #define _ABI_GUC_ERRORS_ABI_H
>>>>
>>>> enum xe_guc_response_status {
>>>> - XE_GUC_RESPONSE_STATUS_SUCCESS = 0x0,
>>>> - XE_GUC_RESPONSE_STATUS_GENERIC_FAIL = 0xF000,
>>>> + XE_GUC_RESPONSE_STATUS_SUCCESS = 0x0,
>>>> + XE_GUC_RESPONSE_NOT_SUPPORTED = 0x20,
>>>> + XE_GUC_RESPONSE_NO_ATTRIBUTE_TABLE = 0x201,
>>>> + XE_GUC_RESPONSE_NO_DECRYPTION_KEY = 0x202,
>>>> + XE_GUC_RESPONSE_DECRYPTION_FAILED = 0x204,
>>>> + XE_GUC_RESPONSE_STATUS_GENERIC_FAIL = 0xF000,
>>>> };
>>>>
>>>> enum xe_guc_load_status {
>>>> @@ -17,6 +21,9 @@ enum xe_guc_load_status {
>>>> XE_GUC_LOAD_STATUS_ERROR_DEVID_BUILD_MISMATCH = 0x02,
>>>> XE_GUC_LOAD_STATUS_GUC_PREPROD_BUILD_MISMATCH = 0x03,
>>>> XE_GUC_LOAD_STATUS_ERROR_DEVID_INVALID_GUCTYPE = 0x04,
>>>> + XE_GUC_LOAD_STATUS_HWCONFIG_START = 0x05,
>>>> + XE_GUC_LOAD_STATUS_HWCONFIG_DONE = 0x06,
>>>> + XE_GUC_LOAD_STATUS_HWCONFIG_ERROR = 0x07,
>>>> XE_GUC_LOAD_STATUS_GDT_DONE = 0x10,
>>>> XE_GUC_LOAD_STATUS_IDT_DONE = 0x20,
>>>> XE_GUC_LOAD_STATUS_LAPIC_DONE = 0x30,
>>>> @@ -34,4 +41,19 @@ enum xe_guc_load_status {
>>>> XE_GUC_LOAD_STATUS_READY = 0xF0,
>>>> };
>>>>
>>>> +enum xe_bootrom_load_status {
>>>> + XE_BOOTROM_STATUS_NO_KEY_FOUND = 0x13,
>>>> + XE_BOOTROM_STATUS_AES_PROD_KEY_FOUND = 0x1A,
>>>> + XE_BOOTROM_STATUS_PROD_KEY_CHECK_FAILURE = 0x2B,
>>>> + XE_BOOTROM_STATUS_RSA_FAILED = 0x50,
>>>> + XE_BOOTROM_STATUS_PAVPC_FAILED = 0x73,
>>>> + XE_BOOTROM_STATUS_WOPCM_FAILED = 0x74,
>>>> + XE_BOOTROM_STATUS_LOADLOC_FAILED = 0x75,
>>>> + XE_BOOTROM_STATUS_JUMP_PASSED = 0x76,
>>>> + XE_BOOTROM_STATUS_JUMP_FAILED = 0x77,
>>>> + XE_BOOTROM_STATUS_RC6CTXCONFIG_FAILED = 0x79,
>>>> + XE_BOOTROM_STATUS_MPUMAP_INCORRECT = 0x7A,
>>>> + XE_BOOTROM_STATUS_EXCEPTION = 0x7E,
>>>> +};
>>>> +
>>>> #endif
>>>> diff --git a/drivers/gpu/drm/xe/regs/xe_guc_regs.h
>>>> b/drivers/gpu/drm/xe/regs/xe_guc_regs.h
>>>> index 92320bbc9d3d..a30e179e662e 100644
>>>> --- a/drivers/gpu/drm/xe/regs/xe_guc_regs.h
>>>> +++ b/drivers/gpu/drm/xe/regs/xe_guc_regs.h
>>>> @@ -40,6 +40,8 @@
>>>> #define GS_BOOTROM_JUMP_PASSED REG_FIELD_PREP(GS_BOOTROM_MASK, 0x76)
>>>> #define GS_MIA_IN_RESET REG_BIT(0)
>>>>
>>>> +#define GUC_HEADER_INFO XE_REG(0xc014)
>>>> +
>>>> #define GUC_WOPCM_SIZE XE_REG(0xc050)
>>>> #define GUC_WOPCM_SIZE_MASK REG_GENMASK(31, 12)
>>>> #define GUC_WOPCM_SIZE_LOCKED REG_BIT(0)
>>>> diff --git a/drivers/gpu/drm/xe/xe_guc.c b/drivers/gpu/drm/xe/xe_guc.c
>>>> index 868208a39829..82514d395704 100644
>>>> --- a/drivers/gpu/drm/xe/xe_guc.c
>>>> +++ b/drivers/gpu/drm/xe/xe_guc.c
>>>> @@ -16,6 +16,7 @@
>>>> #include "xe_device.h"
>>>> #include "xe_force_wake.h"
>>>> #include "xe_gt.h"
>>>> +#include "xe_gt_freq.h"
>>>> #include "xe_guc_ads.h"
>>>> #include "xe_guc_ct.h"
>>>> #include "xe_guc_hwconfig.h"
>>>> @@ -427,58 +428,172 @@ static int guc_xfer_rsa(struct xe_guc *guc)
>>>> return 0;
>>>> }
>>>>
>>>> +/*
>>>> + * Read the GuC status register (GUC_STATUS) and store it in the
>>>> + * specified location; then return a boolean indicating whether
>>>> + * the value matches either completion or a known failure code.
>>>> + *
>>>> + * This is used for polling the GuC status in an xe_wait_for()
>>>> + * loop below.
>>>> + */
>>>> +static inline bool guc_load_done(struct xe_gt *gt, u32 *status,
>>>> bool *success)
>>>
>>> bogus inline
>>>
>>>> +{
>>>> + u32 val = xe_mmio_read32(gt, GUC_STATUS);
>>>> + u32 uk_val = REG_FIELD_GET(GS_UKERNEL_MASK, val);
>>>> + u32 br_val = REG_FIELD_GET(GS_BOOTROM_MASK, val);
>>>> +
>>>> + *status = val;
>>>> + switch (uk_val) {
>>>> + case XE_GUC_LOAD_STATUS_READY:
>>>> + *success = true;
>>>> + return true;
>>>> +
>>>> + case XE_GUC_LOAD_STATUS_ERROR_DEVID_BUILD_MISMATCH:
>>>> + case XE_GUC_LOAD_STATUS_GUC_PREPROD_BUILD_MISMATCH:
>>>> + case XE_GUC_LOAD_STATUS_ERROR_DEVID_INVALID_GUCTYPE:
>>>> + case XE_GUC_LOAD_STATUS_HWCONFIG_ERROR:
>>>> + case XE_GUC_LOAD_STATUS_DPC_ERROR:
>>>> + case XE_GUC_LOAD_STATUS_EXCEPTION:
>>>> + case XE_GUC_LOAD_STATUS_INIT_DATA_INVALID:
>>>> + case XE_GUC_LOAD_STATUS_MPU_DATA_INVALID:
>>>> + case XE_GUC_LOAD_STATUS_INIT_MMIO_SAVE_RESTORE_INVALID:
>>>> + *success = false;
>>>> + return true;
>>>> + }
>>>> +
>>>> + switch (br_val) {
>>>> + case XE_BOOTROM_STATUS_NO_KEY_FOUND:
>>>> + case XE_BOOTROM_STATUS_RSA_FAILED:
>>>> + case XE_BOOTROM_STATUS_PAVPC_FAILED:
>>>> + case XE_BOOTROM_STATUS_WOPCM_FAILED:
>>>> + case XE_BOOTROM_STATUS_LOADLOC_FAILED:
>>>> + case XE_BOOTROM_STATUS_JUMP_FAILED:
>>>> + case XE_BOOTROM_STATUS_RC6CTXCONFIG_FAILED:
>>>> + case XE_BOOTROM_STATUS_MPUMAP_INCORRECT:
>>>> + case XE_BOOTROM_STATUS_EXCEPTION:
>>>> + case XE_BOOTROM_STATUS_PROD_KEY_CHECK_FAILURE:
>>>> + *success = false;
>>>> + return true;
>>>> + }
>>>> +
>>>> + return false;
>>>> +}
>>>> +
>>>> +/*
>>>> + * Wait for the GuC to start up.
>>>> + *
>>>> + * Measurements indicate this should take no more than 20ms
>>>> (assuming the GT
>>>> + * clock is at maximum frequency). However, thermal throttling and
>>>> other issues
>>>> + * can prevent the clock hitting max and thus making the load take
>>>> significantly
>>>> + * longer. Indeed, if the GT is clamped to minimum frequency then
>>>> the load times
>>>> + * can be in the seconds range. As, there is a limit on how long
>>>> an individual
>>>> + * usleep_range() can wait for, the wait is wrapped in a loop. The
>>>> loop count
>>>> + * is increased for debug builds so that problems can be detected
>>>> and analysed.
>>>> + * For release builds, the timeout is kept short so that user's
>>>> don't wait
>>>> + * forever to find out there is a problem. In either case, if the
>>>> load took longer
>>>> + * than is reasonable even with some 'sensible' throttling, then
>>>> flag a warning
>>>> + * because something is not right.
>>>> + *
>>>> + * Note that the only reason an end user should hit the timeout is
>>>> in case of
>>>> + * extreme thermal throttling. And a system that is that hot
>>>> during boot is
>>>> + * probably dead anyway!
>>>> + */
>>>> +#if defined(CONFIG_DRM_XE_DEBUG)
>>>> +#define GUC_LOAD_RETRY_LIMIT 20
>>>> +#else
>>>> +#define GUC_LOAD_RETRY_LIMIT 3
>>>
>>> why? so developers don't reproduce the issues happening on normal
>>> system?
>> Not sure I follow.
>>
>> For CI runs, we want to cope with as much as possible. Anything above
>> the limit below will be flagged as a CI failure, but if a load were
>> to take 4 seconds then having the driver actually complete the load
>> and keep going to run further testing is better than it aborting the
>> load and killing the entire CI run. Especially given Xe's current
>> penchant for causing kernel panics if something fails to start
>> correctly.
>>
>> Whereas, for end users, we want a timeout that is short enough for
>> them to not reach for the power button because their system has hung.
>> As noted, the load should never be in the seconds range unless
>> something is really badly wrong. But that's still not something we
>> want to force on an end user.
>
>
> The problem I have with this thinking is that I don't want CI passing
> and then failing for end users. If CI is completely blocked because our
> timeout wasn't enough, then let it explode so we can fix it. Unless
> there's a reason (e.g. slower machine / environment / etc) for a longer
> timeout/retry we shouldn't make it 8x more just for passing CI.
> If the justification was that "in CI we enable a lot of other debug
> stuff x, y, z that impact this", then it could be acceptable. But then
> the ifdef could also be about those other things. One that comes to mind
> is kasan.
>
> Lucas De Marchi
The point is not to make CI pass. The point is to get a CI failure that
says "GuC took forever to load, GT freq was abysmal, go fix your PCode
bug or replace the fan on the CI system" as opposed to a CI failure that
says "GuC failed to load, no clue why, please waste many days trying to
repro and debug". And to not totally abort the driver load so that
nothing else can be tested. So in the case where it is just a PCODE bug
that only manifests during initial driver load (e.g. uninitialised data
at start of day), testing continues and the whole run is not a total waste.
Note that the 'excessive init time' message is a warn not a dbg. It will
cause a CI failure.
John.
More information about the Intel-xe
mailing list