<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word"> <p style="font-family:Arial;font-size:10pt;color:#0000FF;margin:5pt;font-style:normal;font-weight:normal;text-decoration:none;" align="Left"> [AMD Official Use Only - General]<br> </p> <br> <div> <div class="WordSection1"> <p class="MsoNormal">I just make one debug patch to show busy BO’s alloc-trace when the eviction fails in suspend.<o:p></o:p></p> <p class="MsoNormal">And dmesg log attached.<o:p></o:p></p> <p class="MsoNormal">Looks like they are just kfd user Bos and locked by evict/restore work.<o:p></o:p></p> <p class="MsoNormal">So in kfd suspend callback, it really need to flush the evict/restore work before HW fini as it do now.<o:p></o:p></p> <p class="MsoNormal">That is why the first very early eviction fails and the second eviction succeed.<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Thanks<o:p></o:p></p> <p class="MsoNormal">xinhui<o:p></o:p></p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal"><b>From:</b> Pan, Xinhui <br> <b>Sent:</b> Thursday, September 14, 2023 8:02 AM<br> <b>To:</b> Koenig, Christian <Christian.Koenig@amd.com>; Kuehling, Felix <Felix.Kuehling@amd.com>; Christian König <ckoenig.leichtzumerken@gmail.com>; amd-gfx@lists.freedesktop.org; Wentland, Harry <Harry.Wentland@amd.com><br> <b>Cc:</b> Deucher, Alexander <Alexander.Deucher@amd.com>; Fan, Shikang <Shikang.Fan@amd.com><br> <b>Subject:</b> RE: <span lang="ZH-CN" style="font-family:"Microsoft YaHei",sans-serif"> 回复</span>: [PATCH] drm/amdgpu: Ignore first evction failure during suspend<o:p></o:p></p> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Chris,<o:p></o:p></p> <p class="MsoNormal">I can dump these busy BOs with their alloc/free stack later today.<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">BTW, the two evictions and the kfd suspend are all called before hw_fini. IOW, between phase 1 and phase 2. SDMA is turned only in phase2. So current code works fine maybe.<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p class="MsoNormal"><b>From:</b> Koenig, Christian <<a href="mailto:Christian.Koenig@amd.com">Christian.Koenig@amd.com</a>> <br> <b>Sent:</b> Wednesday, September 13, 2023 10:29 PM<br> <b>To:</b> Kuehling, Felix <<a href="mailto:Felix.Kuehling@amd.com">Felix.Kuehling@amd.com</a>>; Christian König <<a href="mailto:ckoenig.leichtzumerken@gmail.com">ckoenig.leichtzumerken@gmail.com</a>>; Pan, Xinhui <<a href="mailto:Xinhui.Pan@amd.com">Xinhui.Pan@amd.com</a>>; <a href="mailto:amd-gfx@lists.freedesktop.org">amd-gfx@lists.freedesktop.org</a>; Wentland, Harry <<a href="mailto:Harry.Wentland@amd.com">Harry.Wentland@amd.com</a>><br> <b>Cc:</b> Deucher, Alexander <<a href="mailto:Alexander.Deucher@amd.com">Alexander.Deucher@amd.com</a>>; Fan, Shikang <<a href="mailto:Shikang.Fan@amd.com">Shikang.Fan@amd.com</a>><br> <b>Subject:</b> Re: <span lang="ZH-CN" style="font-family:"Microsoft YaHei",sans-serif"> 回复</span>: [PATCH] drm/amdgpu: Ignore first evction failure during suspend<o:p></o:p></p> </div> </div> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal" style="margin-bottom:12.0pt">[+Harry]<o:p></o:p></p> <div> <p class="MsoNormal">Am 13.09.23 um 15:54 schrieb Felix Kuehling:<o:p></o:p></p> </div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <div> <p class="MsoNormal">On 2023-09-13 4:07, Christian König wrote:<o:p></o:p></p> </div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <p class="MsoNormal">[+Fleix]<br> <br> Well that looks like quite a serious bug.<br> <br> If I'm not completely mistaken the KFD work item tries to restore the process by moving BOs into memory even after the suspend freeze. Normally work items are frozen together with the user space processes unless explicitly marked as not freezable.<br> <br> That this causes problem during the first eviction phase is just the tip of the iceberg here. If a BO is moved into invisible memory during this we wouldn't be able to get it out of that in the second phase because SDMA and hw is already turned off.<br> <br> @Felix any idea how that can happen? Have you guys marked a work item / work queue as not freezable?<o:p></o:p></p> </blockquote> <p>We don't set anything to non-freezable in KFD.<o:p></o:p></p> <p><o:p> </o:p></p> <p>Regards,<br> Felix<o:p></o:p></p> <p><o:p> </o:p></p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <p class="MsoNormal">Or maybe the display guys?<o:p></o:p></p> </blockquote> </blockquote> <p class="MsoNormal" style="margin-bottom:12.0pt"><br> Do you guys in the display do any delayed update in a work item which is marked as not-freezable?<br> <br> Otherwise I have absolutely no idea what's going on here.<br> <br> Thanks,<br> Christian.<br> <br> <o:p></o:p></p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <p class="MsoNormal" style="margin-bottom:12.0pt"><br> @Xinhui please investigate what work item that is and where that is coming from. Something like "if (adev->in_suspend) dump_stack();" in the right place should probably do it.<br> <br> Thanks,<br> Christian.<o:p></o:p></p> <div> <p class="MsoNormal">Am 13.09.23 um 07:13 schrieb Pan, Xinhui:<o:p></o:p></p> </div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <p style="margin:5.0pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:blue">[AMD Official Use Only - General]<o:p></o:p></span></p> <p class="MsoNormal"><o:p> </o:p></p> <div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">I notice that only user space process are frozen on my side. kthread and workqueue keeps running. Maybe some kernel configs are not enabled.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">I made one module which just prints something like i++ with mutex lock both in workqueue and kthread. I paste some logs below.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438619.696196] XH: 14 from workqueue <o:p></o:p></span></p> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438619.700193] XH: 15 from kthread<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.394335] PM: suspend entry (deep)<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.399619] Filesystems sync: 0.001 seconds<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.403887] PM: Preparing system for sleep (deep)<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.409299] Freezing user space processes<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.414862] Freezing user space processes completed (elapsed 0.001 seconds)<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.421881] OOM killer disabled.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.425197] Freezing remaining freezable tasks<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.430890] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)<o:p></o:p></span></p> </div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.438348] PM: Suspending system (deep)<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">.....<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438623.746038] PM: suspend of devices complete after 3303.137 msecs <o:p></o:p></span></p> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438623.752125] PM: start suspend of devices complete after 3309.713 msecs<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438623.758722] PM: suspend debug: Waiting for 5 second(s).<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438623.792166] XH: 22 from kthread<o:p></o:p></span></p> </div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438623.824140] XH: 23 from workqueue<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">So BOs definitely can be in use during suspend.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">Even if kthread or workqueue can be stopped with one special kernel config. I think suspend can only stop the workqueue with its callback finish. <o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">otherwise something like below makes things crazy.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">LOCK BO<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">do something<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"> -> schedule or wait, anycode might sleep. Stopped by suspend now? no, i think.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">UNLOCK BO<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">I do tests with cmds below.<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">echo devices > /sys/power/pm_test <o:p></o:p></span></p> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">echo 0 > /sys/power/pm_async<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">echo 1 > /sys/power/pm_print_times<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">echo 1 > /sys/power/pm_debug_messages<o:p></o:p></span></p> </div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">echo 1 > /sys/module/amdgpu/parameters/debug_evictions<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">./kfd.sh --gtest_filter=KFDEvictTest.BasicTest<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">pm-suspend<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">thanks<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">xinhui<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p> </div> <div class="MsoNormal" align="center" style="text-align:center"> <hr size="1" width="98%" align="center"> </div> <div id="divRplyFwdMsg"> <p class="MsoNormal"><b><span lang="ZH-CN" style="font-family:"Microsoft YaHei",sans-serif;color:black">发件人</span><span style="color:black">:</span></b><span style="color:black"> Christian König <a href="mailto:ckoenig.leichtzumerken@gmail.com"><ckoenig.leichtzumerken@gmail.com></a><br> </span><b><span lang="ZH-CN" style="font-family:"Microsoft YaHei",sans-serif;color:black">发送时间</span><span style="color:black">:</span></b><span style="color:black"> 2023</span><span lang="ZH-CN" style="font-family:"Microsoft YaHei",sans-serif;color:black">年</span><span style="color:black">9</span><span lang="ZH-CN" style="font-family:"Microsoft YaHei",sans-serif;color:black">月</span><span style="color:black">12</span><span lang="ZH-CN" style="font-family:"Microsoft YaHei",sans-serif;color:black">日</span><span style="color:black"> 17:01<br> </span><b><span lang="ZH-CN" style="font-family:"Microsoft YaHei",sans-serif;color:black">收件人</span><span style="color:black">:</span></b><span style="color:black"> Pan, Xinhui <a href="mailto:Xinhui.Pan@amd.com"><Xinhui.Pan@amd.com></a>; <a href="mailto:amd-gfx@lists.freedesktop.org"> amd-gfx@lists.freedesktop.org</a> <a href="mailto:amd-gfx@lists.freedesktop.org"> <amd-gfx@lists.freedesktop.org></a><br> </span><b><span lang="ZH-CN" style="font-family:"Microsoft YaHei",sans-serif;color:black">抄送</span><span style="color:black">:</span></b><span style="color:black"> Deucher, Alexander <a href="mailto:Alexander.Deucher@amd.com"><Alexander.Deucher@amd.com></a>; Koenig, Christian <a href="mailto:Christian.Koenig@amd.com"><Christian.Koenig@amd.com></a>; Fan, Shikang <a href="mailto:Shikang.Fan@amd.com"><Shikang.Fan@amd.com></a><br> </span><b><span lang="ZH-CN" style="font-family:"Microsoft YaHei",sans-serif;color:black">主题</span><span style="color:black">:</span></b><span style="color:black"> Re: [PATCH] drm/amdgpu: Ignore first evction failure during suspend</span> <o:p></o:p></p> <div> <p class="MsoNormal"> <o:p></o:p></p> </div> </div> <div> <div> <p class="MsoNormal" style="margin-bottom:12.0pt">When amdgpu_device_suspend() is called processes should be frozen <br> already. In other words KFD queues etc... should already be idle.<br> <br> So when the eviction fails here we missed something previously and that <br> in turn can cause tons amount of problems.<br> <br> So ignoring those errors is most likely not a good idea at all.<br> <br> Regards,<br> Christian.<br> <br> Am 12.09.23 um 02:21 schrieb Pan, Xinhui:<br> > [AMD Official Use Only - General]<br> ><br> > Oh yep, Pinned BO is moved to other LRU list, So eviction fails because of other reason.<br> > I will change the comments in the patch.<br> > The problem is eviction fails as many reasons, say, BO is locked.<br> > ASAIK, kfd will stop the queues and flush some evict/restore work in its suspend callback. SO the first eviction before kfd callback likely fails.<br> ><br> > -----Original Message-----<br> > From: Christian König <a href="mailto:ckoenig.leichtzumerken@gmail.com"><ckoenig.leichtzumerken@gmail.com></a><br> > Sent: Friday, September 8, 2023 2:49 PM<br> > To: Pan, Xinhui <a href="mailto:Xinhui.Pan@amd.com"><Xinhui.Pan@amd.com></a>; <a href="mailto:amd-gfx@lists.freedesktop.org"> amd-gfx@lists.freedesktop.org</a><br> > Cc: Deucher, Alexander <a href="mailto:Alexander.Deucher@amd.com"><Alexander.Deucher@amd.com></a>; Koenig, Christian <a href="mailto:Christian.Koenig@amd.com"><Christian.Koenig@amd.com></a>; Fan, Shikang <a href="mailto:Shikang.Fan@amd.com"><Shikang.Fan@amd.com></a><br> > Subject: Re: [PATCH] drm/amdgpu: Ignore first evction failure during suspend<br> ><br> > Am 08.09.23 um 05:39 schrieb xinhui pan:<br> >> Some BOs might be pinned. So the first eviction's failure will abort<br> >> the suspend sequence. These pinned BOs will be unpined afterwards<br> >> during suspend.<br> > That doesn't make much sense since pinned BOs don't cause eviction failure here.<br> ><br> > What exactly is the error code you see?<br> ><br> > Christian.<br> ><br> >> Actaully it has evicted most BOs, so that should stil work fine in<br> >> sriov full access mode.<br> >><br> >> Fixes: 47ea20762bb7 ("drm/amdgpu: Add an extra evict_resource call<br> >> during device_suspend.")<br> >> Signed-off-by: xinhui pan <a href="mailto:xinhui.pan@amd.com"><xinhui.pan@amd.com></a><br> >> ---<br> >> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 9 +++++----<br> >> 1 file changed, 5 insertions(+), 4 deletions(-)<br> >><br> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br> >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br> >> index 5c0e2b766026..39af526cdbbe 100644<br> >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br> >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br> >> @@ -4148,10 +4148,11 @@ int amdgpu_device_suspend(struct drm_device<br> >> *dev, bool fbcon)<br> >><br> >> adev->in_suspend = true;<br> >><br> >> - /* Evict the majority of BOs before grabbing the full access */<br> >> - r = amdgpu_device_evict_resources(adev);<br> >> - if (r)<br> >> - return r;<br> >> + /* Try to evict the majority of BOs before grabbing the full access<br> >> + * Ignore the ret val at first place as we will unpin some BOs if any<br> >> + * afterwards.<br> >> + */<br> >> + (void)amdgpu_device_evict_resources(adev);<br> >><br> >> if (amdgpu_sriov_vf(adev)) {<br> >> amdgpu_virt_fini_data_exchange(adev);<o:p></o:p></p> </div> </div> </div> </blockquote> <p class="MsoNormal"><o:p> </o:p></p> </blockquote> </blockquote> <p class="MsoNormal"><o:p> </o:p></p> </div> </div> </body> </html>