<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=Windows-1252"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="en-CN" link="blue" vlink="purple" style="word-wrap:break-word"> [AMD Official Use Only - AMD Internal Distribution Only] <div> <div class="WordSection1"> Hi <a id="OWAAMA95337AF4FF1064F84FD15F1EAAF1561" href="mailto:HaiJun.Chang@amd.com"> @Chang, HaiJun</a><o:p></o:p> <o:p> </o:p> Thank you for the info. <o:p></o:p> You are right, GFXMSIX_VECT0_ADDR_LO and GFXMSIX_VECT0_CONTROL registers are not restored on resume with new VF. source VF, normal value<o:p></o:p> 0x42000: 0xFEE00138 // GFXMSIX_VECT0_ADDR_LO<o:p></o:p> 0x4200C: 0x00000000 // GFXMSIX_VECT0_CONTROL<o:p></o:p> <o:p> </o:p> destination VF, abnormal value<o:p></o:p> 0x42000: 0x00000000 // GFXMSIX_VECT0_ADDR_LO<o:p></o:p> 0x4200C: 0x00000001 // GFXMSIX_VECT0_CONTROL<o:p></o:p> <o:p> </o:p> Calling amdgpu_restore_msix() on resume can fix this issue. I will upload a new patch for this. static int vega20_ih_resume(struct amdgpu_ip_block *ip_block)<o:p></o:p> {<o:p></o:p> + struct amdgpu_device *adev = ip_block->adev;<o:p></o:p> +<o:p></o:p> + if (amdgpu_xmgi_is_node_changed(adev))<o:p></o:p> + amdgpu_restore_msix(adev);<o:p></o:p> return vega20_ih_hw_init(ip_block);<o:p></o:p> }<o:p></o:p> <o:p> </o:p> Thanks<o:p></o:p> Sam<o:p></o:p> <o:p> </o:p> <div id="mail-editor-reference-message-container"> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Chang, HaiJun <HaiJun.Chang@amd.com> Date: Tuesday, April 29, 2025 at 10:43 To: Koenig, Christian <Christian.Koenig@amd.com>, Zhang, GuoQing (Sam) <GuoQing.Zhang@amd.com>, Christian König <ckoenig.leichtzumerken@gmail.com>, amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>, Deucher, Alexander <Alexander.Deucher@amd.com> Cc: Zhao, Victor <Victor.Zhao@amd.com>, Deng, Emily <Emily.Deng@amd.com>, Zhang, Owen(SRDC) <Owen.Zhang2@amd.com> Subject: RE: [PATCH 6/6] drm/amdgpu: fix fence fallback timer expired error<o:p></o:p> </div> <div> <a name="BM_BEGIN"></a>[AMD Official Use Only - AMD Internal Distribution Only] Hi, The interrupt issue loss issue might be VF msix table isn't restored properly on resume. The msix table in virtual machine is faked. The real msix table will be programmed by QEMU when guest enable/disable msix interrupt. But QEMU accessing VF msix table (GFXMSIX_* registers) could be blocked by nBIF protection if that time VF isn't in exclusive access. We had a w/a in amdgpu driver to handle msix table loss case in amdgpu_restore_msix function. Can you check the values of these GFXMSIX_* registers? I think we should call amdgpu_restore_msix on resume to restore msix table. Thanks, HaiJun -----Original Message----- From: Koenig, Christian <Christian.Koenig@amd.com> Sent: Monday, April 28, 2025 8:24 PM To: Zhang, GuoQing (Sam) <GuoQing.Zhang@amd.com>; Christian König <ckoenig.leichtzumerken@gmail.com>; amd-gfx@lists.freedesktop.org; Deucher, Alexander <Alexander.Deucher@amd.com> Cc: Zhao, Victor <Victor.Zhao@amd.com>; Chang, HaiJun <HaiJun.Chang@amd.com>; Deng, Emily <Emily.Deng@amd.com>; Zhang, Owen(SRDC) <Owen.Zhang2@amd.com> Subject: Re: [PATCH 6/6] drm/amdgpu: fix fence fallback timer expired error On 4/24/25 05:38, Zhang, GuoQing (Sam) wrote: > [AMD Official Use Only - AMD Internal Distribution Only] > > > Ping… @Koenig, Christian <<a href="mailto:Christian.Koenig@amd.com">mailto:Christian.Koenig@amd.com</a>> > > Thanks > > Sam > > *From: *amd-gfx <amd-gfx-bounces@lists.freedesktop.org> on behalf of > Zhang, GuoQing (Sam) <GuoQing.Zhang@amd.com> > *Date: *Wednesday, April 23, 2025 at 14:59 > *To: *Christian König <ckoenig.leichtzumerken@gmail.com>, amd- > gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org> > *Cc: *Zhao, Victor <Victor.Zhao@amd.com>, Chang, HaiJun > <HaiJun.Chang@amd.com>, Deng, Emily <Emily.Deng@amd.com>, Zhang, > Owen(SRDC) <Owen.Zhang2@amd.com> > *Subject: *Re: [PATCH 6/6] drm/amdgpu: fix fence fallback timer > expired error > > [AMD Official Use Only - AMD Internal Distribution Only] > > [AMD Official Use Only - AMD Internal Distribution Only] > > Hi @Christian König <<a href="mailto:ckoenig.leichtzumerken@gmail.com">mailto:ckoenig.leichtzumerken@gmail.com</a>>, > > On QEMU VM environment, when request_irq() is called in guest KMD, > QEMU will enable interrupt for the device on the host. > > When hibernate and resume with a new vGPU without calling > request_irq() on the new vGPU, the interrupt of the new vGPU is not > enabled. The IH handler in guest KMD will not be called in this case. > > This change is to ensure request_irq() is called on resume for the new vGPUs. That doesn't make sense. The MSI state is saved and restored by the core OS on suspend and resume, drivers should never mess with that. If this doesn't work with the new vGPU for some reason then that is not something we can work around inside the driver. Which state exactly isn't restored here? Regards, Christian. > > Regards > > Sam > > *From: *Christian König <ckoenig.leichtzumerken@gmail.com> > *Date: *Wednesday, April 16, 2025 at 21:54 > *To: *Zhang, GuoQing (Sam) <GuoQing.Zhang@amd.com>, amd- > gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org> > *Cc: *Zhao, Victor <Victor.Zhao@amd.com>, Chang, HaiJun > <HaiJun.Chang@amd.com>, Deng, Emily <Emily.Deng@amd.com> > *Subject: *Re: [PATCH 6/6] drm/amdgpu: fix fence fallback timer > expired error > > Am 14.04.25 um 12:46 schrieb Samuel Zhang: > > IH is not working after switching a new gpu index for the first time. > > IH handler function need to be re-registered with kernel after > switching > to new gpu index. > > Why? > > Christian. > > > > > Signed-off-by: Samuel Zhang <guoqing.zhang@amd.com> > Change-Id: > Idece1c8fce24032fd08f5a8b6ac23793c51e56dd > > --- > > drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 7 +++++-- > > drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h | 1 + > > drivers/gpu/drm/amd/amdgpu/vega20_ih.c | 18 ++++++++++++++++-- > 3 > files changed, 22 insertions(+), 4 deletions(-) > > diff --git > a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c b/drivers/gpu/drm/amd/ > amdgpu/amdgpu_irq.c > index 19ce4da285e8..2292245a0c5d 100644 > --- > a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c > > @@ -326,7 +326,7 @@ int amdgpu_irq_init(struct amdgpu_device *adev) > > return r; > > } > > > > -void amdgpu_irq_fini_hw(struct amdgpu_device *adev) > +void > amdgpu_irq_uninstall(struct amdgpu_device *adev) > { > > if (adev->irq.installed) { > > free_irq(adev->irq.irq, adev_to_drm(adev)); > > @@ -334,7 +334,10 @@ void amdgpu_irq_fini_hw(struct amdgpu_device *adev) > > if (adev->irq.msi_enabled) > > pci_free_irq_vectors(adev->pdev); > > } > > - > > +} > > +void amdgpu_irq_fini_hw(struct amdgpu_device *adev) > +{ > > + amdgpu_irq_uninstall(adev); > > amdgpu_ih_ring_fini(adev, &adev->irq.ih_soft); > > amdgpu_ih_ring_fini(adev, &adev->irq.ih); > > amdgpu_ih_ring_fini(adev, &adev->irq.ih1); > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h > b/drivers/gpu/drm/amd/ amdgpu/amdgpu_irq.h > index > 04c0b4fa17a4..c6e6681b4f71 100644 > --- > a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h > > @@ -123,6 +123,7 @@ extern const int > node_id_to_phys_map[NODEID_MAX]; > void > amdgpu_irq_disable_all(struct amdgpu_device *adev); > > int > amdgpu_irq_init(struct amdgpu_device *adev); > +void > amdgpu_irq_uninstall(struct amdgpu_device *adev); > void > amdgpu_irq_fini_sw(struct amdgpu_device *adev); > void > amdgpu_irq_fini_hw(struct amdgpu_device *adev); > int > amdgpu_irq_add_id(struct amdgpu_device *adev, > diff --git > a/drivers/gpu/drm/amd/amdgpu/vega20_ih.c b/drivers/gpu/drm/amd/ > amdgpu/vega20_ih.c > index faa0dd75dd6d..ef996505e4dc 100644 > --- > a/drivers/gpu/drm/amd/amdgpu/vega20_ih.c > > +++ b/drivers/gpu/drm/amd/amdgpu/vega20_ih.c > > @@ -643,12 +643,26 @@ static int vega20_ih_hw_fini(struct > amdgpu_ip_block > *ip_block) > > > > static int vega20_ih_suspend(struct amdgpu_ip_block *ip_block) > > { > > - return vega20_ih_hw_fini(ip_block); > > + struct amdgpu_device *adev = ip_block->adev; > > + int r = 0; > > + > > + r = vega20_ih_hw_fini(ip_block); > > + amdgpu_irq_uninstall(adev); > > + return r; > > } > > > > static int vega20_ih_resume(struct amdgpu_ip_block *ip_block) > > { > > - return vega20_ih_hw_init(ip_block); > > + struct amdgpu_device *adev = ip_block->adev; > > + int r = 0; > > + > > + r = amdgpu_irq_init(adev); > > + if (r) { > > + dev_err(adev->dev, "amdgpu_irq_init failed in %s, %d\n", > __func__, r); > > + return r; > > + } > > + r = vega20_ih_hw_init(ip_block); > > + return r; > > } > > > > static bool vega20_ih_is_idle(struct amdgpu_ip_block *ip_block) ><o:p></o:p> </div> </div> </div> </div> </div> </body> </html>