[PATCH] drm/amd: Add a workaround for GFX11 systems that fail to flush TLB
Alex Deucher
alexdeucher at gmail.com
Wed Dec 13 19:44:42 UTC 2023
On Wed, Dec 13, 2023 at 2:32 PM Mario Limonciello
<mario.limonciello at amd.com> wrote:
>
> On 12/13/2023 13:12, Mario Limonciello wrote:
> > On 12/13/2023 13:07, Alex Deucher wrote:
> >> On Wed, Dec 13, 2023 at 1:00 PM Mario Limonciello
> >> <mario.limonciello at amd.com> wrote:
> >>>
> >>> Some systems with MP1 13.0.4 or 13.0.11 have a firmware bug that
> >>> causes the first MES packet after resume to fail. This packet is
> >>> used to flush the TLB when GART is enabled.
> >>>
> >>> This issue is fixed in newer firmware, but as OEMs may not roll this
> >>> out to the field, introduce a workaround that will retry the flush
> >>> when detecting running on an older firmware and decrease relevant
> >>> error messages to debug while workaround is in use.
> >>>
> >>> Cc: stable at vger.kernel.org # 6.1+
> >>> Cc: Tim Huang <Tim.Huang at amd.com>
> >>> Closes: https://gitlab.freedesktop.org/drm/amd/-/issues/3045
> >>> Signed-off-by: Mario Limonciello <mario.limonciello at amd.com>
> >>> ---
> >>> drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c | 10 ++++++++--
> >>> drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h | 2 ++
> >>> drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c | 17 ++++++++++++++++-
> >>> drivers/gpu/drm/amd/amdgpu/mes_v11_0.c | 8 ++++++--
> >>> 4 files changed, 32 insertions(+), 5 deletions(-)
> >>>
> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
> >>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
> >>> index 9ddbf1494326..6ce3f6e6b6de 100644
> >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
> >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c
> >>> @@ -836,8 +836,14 @@ int amdgpu_mes_reg_write_reg_wait(struct
> >>> amdgpu_device *adev,
> >>> }
> >>>
> >>> r = adev->mes.funcs->misc_op(&adev->mes, &op_input);
> >>> - if (r)
> >>> - DRM_ERROR("failed to reg_write_reg_wait\n");
> >>> + if (r) {
> >>> + const char *msg = "failed to reg_write_reg_wait\n";
> >>> +
> >>> + if (adev->mes.suspend_workaround)
> >>> + DRM_DEBUG(msg);
> >>> + else
> >>> + DRM_ERROR(msg);
> >>> + }
> >>>
> >>> error:
> >>> return r;
> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h
> >>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h
> >>> index a27b424ffe00..90f2bba3b12b 100644
> >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h
> >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h
> >>> @@ -135,6 +135,8 @@ struct amdgpu_mes {
> >>>
> >>> /* ip specific functions */
> >>> const struct amdgpu_mes_funcs *funcs;
> >>> +
> >>> + bool suspend_workaround;
> >>> };
> >>>
> >>> struct amdgpu_mes_process {
> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c
> >>> b/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c
> >>> index 23d7b548d13f..e810c7bb3156 100644
> >>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c
> >>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c
> >>> @@ -889,7 +889,11 @@ static int gmc_v11_0_gart_enable(struct
> >>> amdgpu_device *adev)
> >>> false : true;
> >>>
> >>> adev->mmhub.funcs->set_fault_enable_default(adev, value);
> >>> - gmc_v11_0_flush_gpu_tlb(adev, 0, AMDGPU_MMHUB0(0), 0);
> >>> +
> >>> + do {
> >>> + gmc_v11_0_flush_gpu_tlb(adev, 0, AMDGPU_MMHUB0(0), 0);
> >>> + adev->mes.suspend_workaround = false;
> >>> + } while (adev->mes.suspend_workaround);
> >>
> >> Shouldn't this be something like:
> >>
> >>> + do {
> >>> + gmc_v11_0_flush_gpu_tlb(adev, 0, AMDGPU_MMHUB0(0), 0);
> >>> + adev->mes.suspend_workaround = false;
> >>> + gmc_v11_0_flush_gpu_tlb(adev, 0, AMDGPU_MMHUB0(0), 0);
> >>> + } while (adev->mes.suspend_workaround);
> >>
> >> If we actually need the flush. Maybe a better approach would be to
> >> check if we are in s0ix in
> >
> > Ah you're right; I had shifted this around to keep less stateful
> > variables and push them up the stack from when I first made it and that
> > logic is wrong now.
> >
> > I don't think the one you suggested is right either; it's going to apply
> > twice on ASICs that only need it once.
> >
> > I guess pending on what Christian comments on below I'll respin to logic
> > that only calls twice on resume for these ASICs.
>
> One more comment. Tim and I both did an experiment for this (skipping
> the flush) separately. The problem isn't the flush itself, rather it's
> the first MES packet after exiting GFXOFF.
>
> So it seems that it pushes off the issue to the next thing which is a
> ring buffer test:
>
> [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.0.0
> (-110).
> [drm:process_one_work] *ERROR* ib ring test failed (-110).
>
> So maybe a better workaround is a "dummy" command that is only sent for
> the broken firmware that we don't care about the outcome and discard errors.
>
> Then the workaround doesn't need to get as entangled with correct flow.
Yeah. Something like that seems cleaner. Just a question of where to
put it since we skip GC and MES for s0ix. Probably somewhere in
gmc_v11_0_resume() before gmc_v11_0_gart_enable(). Maybe add a new
mes callback.
Alex
>
> >
> >>
> >> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c
> >> b/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c in gmc_v11_0_flush_gpu_tlb():
> >> index 23d7b548d13f..bd6d9953a80e 100644
> >> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c
> >> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c
> >> @@ -227,7 +227,8 @@ static void gmc_v11_0_flush_gpu_tlb(struct
> >> amdgpu_device *adev, uint32_t vmid,
> >> * Directly use kiq to do the vm invalidation instead
> >> */
> >> if ((adev->gfx.kiq[0].ring.sched.ready ||
> >> adev->mes.ring.sched.ready) &&
> >> - (amdgpu_sriov_runtime(adev) || !amdgpu_sriov_vf(adev))) {
> >> + (amdgpu_sriov_runtime(adev) || !amdgpu_sriov_vf(adev)) ||
> >> + !adev->in_s0ix) {
> >> amdgpu_virt_kiq_reg_write_reg_wait(adev, req, ack,
> >> inv_req,
> >> 1 << vmid, GET_INST(GC, 0));
> >> return;
> >>
> >> @Christian Koenig is this logic correct?
> >>
> >> /* For SRIOV run time, driver shouldn't access the register
> >> through MMIO
> >> * Directly use kiq to do the vm invalidation instead
> >> */
> >> if ((adev->gfx.kiq[0].ring.sched.ready ||
> >> adev->mes.ring.sched.ready) &&
> >> (amdgpu_sriov_runtime(adev) || !amdgpu_sriov_vf(adev))) {
> >> amdgpu_virt_kiq_reg_write_reg_wait(adev, req, ack,
> >> inv_req,
> >> 1 << vmid, GET_INST(GC, 0));
> >> return;
> >> }
> >>
> >> We basically always use the MES with that logic. If that is the case,
> >> we should just drop the rest of that function. Shouldn't we only use
> >> KIQ or MES for SR-IOV? gmc v10 has similar logic which also seems
> >> wrong.
> >>
> >> Alex
> >>
> >>
> >>>
> >>> DRM_INFO("PCIE GART of %uM enabled (table at 0x%016llX).\n",
> >>> (unsigned int)(adev->gmc.gart_size >> 20),
> >>> @@ -960,6 +964,17 @@ static int gmc_v11_0_resume(void *handle)
> >>> int r;
> >>> struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> >>>
> >>> + switch (amdgpu_ip_version(adev, MP1_HWIP, 0)) {
> >>> + case IP_VERSION(13, 0, 4):
> >>> + case IP_VERSION(13, 0, 11):
> >>> + /* avoid problems with first TLB flush after resume */
> >>> + if ((adev->pm.fw_version & 0x00FFFFFF) < 0x004c4900)
> >>> + adev->mes.suspend_workaround = adev->in_s0ix;
> >>> + break;
> >>> + default:
> >>> + break;
> >>> + }
> >>> +
> >>> r = gmc_v11_0_hw_init(adev);
> >>> if (r)
> >>> return r;
> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
> >>> b/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
> >>> index 4dfec56e1b7f..84ab8c611e5e 100644
> >>> --- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
> >>> +++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c
> >>> @@ -137,8 +137,12 @@ static int
> >>> mes_v11_0_submit_pkt_and_poll_completion(struct amdgpu_mes *mes,
> >>> r = amdgpu_fence_wait_polling(ring, ring->fence_drv.sync_seq,
> >>> timeout);
> >>> if (r < 1) {
> >>> - DRM_ERROR("MES failed to response msg=%d\n",
> >>> - x_pkt->header.opcode);
> >>> + if (mes->suspend_workaround)
> >>> + DRM_DEBUG("MES failed to response msg=%d\n",
> >>> + x_pkt->header.opcode);
> >>> + else
> >>> + DRM_ERROR("MES failed to response msg=%d\n",
> >>> + x_pkt->header.opcode);
> >>>
> >>> while (halt_if_hws_hang)
> >>> schedule();
> >>> --
> >>> 2.34.1
> >>>
> >
>
More information about the amd-gfx
mailing list