<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
Am 14.09.23 um 15:37 schrieb Felix Kuehling:<br>
<blockquote type="cite" cite="mid:c4b952bf-c8c0-10de-e168-61ee16790c81@amd.com">
<p>Userptr and SVM restore work is scheduled to the system WQ with
schedule_delayed_work. See amdgpu_amdkfd_evict_userptr and
svm_range_evict. This would need to use queue_delayed_work with
the system_freezable_wq.<br>
</p>
<p>BO restoration is scheduled with queue_delayed_work on our own
kfd_restore_wq that was allocated with alloc_ordered_workqueue.
This would need to add the WQ_FREEZABLE flag when we create the
wq in kfd_process_create_wq.<br>
</p>
<p>There is also evict_process_worker scheduled with
schedule_delayed_work, which handles stopping of user mode
queues, signaling of eviction fences and scheduling of restore
work when BOs are evicted. I think that should not be freezable
because it's needed to signal the eviction fences to allow
suspend to evict BOs.<br>
</p>
<p>To make sure I'm not misunderstanding, I assume that freezing a
freezable workqueue flushes work items in progress and prevents
execution of more work until it is unfrozen. I assume work items
are not frozen in the middle of execution, because that would
not solve the problem.<br>
</p>
</blockquote>
<br>
I was wondering the exact same thing and to be honest I don't know
that detail either and of hand can't find any documentation about
it.<br>
<br>
My suspicion is that a work item can freeze when it calls
schedule(), e.g. when taking a look or similar.<br>
<br>
That would then indeed not work at all and we would need to make
sure that the work is completed manually somehow.<br>
<br>
Regards,<br>
Christian.<br>
<br>
<blockquote type="cite" cite="mid:c4b952bf-c8c0-10de-e168-61ee16790c81@amd.com">
<p> </p>
<p>Regards,<br>
Felix</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 2023-09-14 2:23, Christian König
wrote:<br>
</div>
<blockquote type="cite" cite="mid:2e2c730d-f8f2-cda7-74cb-91b493da8902@gmail.com">
[putting Harry on BCC, sorry for the noise]<br>
<br>
Yeah, that is clearly a bug in the KFD.<br>
<br>
During the second eviction the hw should already be disabled, so
we don't have any SDMA or similar to evict BOs any more and can
only copy them with the CPU.<br>
<br>
@Felix what workqueue do you guys use for the restore work? I've
just double checked and on the system workqueues you explicitly
need to specify that stuff is freezable. E.g. use
system_freezable_wq instead of system_wq.<br>
<br>
Alternatively as Xinhui mentioned it might be necessary to flush
all restore work before the first eviction phase or we have the
chance that BOs are moved back into VRAM again.<br>
<br>
Regards,<br>
Christian.<br>
<br>
<div class="moz-cite-prefix">Am 14.09.23 um 03:54 schrieb Pan,
Xinhui:<br>
</div>
<blockquote type="cite" cite="mid:DM4PR12MB5165B9687FBB14DAEF11944187F7A@DM4PR12MB5165.namprd12.prod.outlook.com">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}@font-face
{font-family:"Microsoft YaHei";
panose-1:2 11 5 3 2 2 4 2 2 4;}@font-face
{font-family:"\@Microsoft YaHei";}@font-face
{font-family:Aptos;}@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}div.WordSection1
{page:WordSection1;}</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<p style="font-family:Arial;font-size:10pt;color:#0000FF;margin:5pt;font-style:normal;font-weight:normal;text-decoration:none;" align="Left"> [AMD Official Use Only - General]<br>
</p>
<br>
<div>
<div class="WordSection1">
<p class="MsoNormal">I just make one debug patch to show
busy BO’s alloc-trace when the eviction fails in
suspend.<o:p></o:p></p>
<p class="MsoNormal">And dmesg log attached.<o:p></o:p></p>
<p class="MsoNormal">Looks like they are just kfd user Bos
and locked by evict/restore work.<o:p></o:p></p>
<p class="MsoNormal">So in kfd suspend callback, it really
need to flush the evict/restore work before HW fini as
it do now.<o:p></o:p></p>
<p class="MsoNormal">That is why the first very early
eviction fails and the second eviction succeed.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal">xinhui<o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Pan, Xinhui <br>
<b>Sent:</b> Thursday, September 14, 2023 8:02 AM<br>
<b>To:</b> Koenig, Christian <a class="moz-txt-link-rfc2396E" href="mailto:Christian.Koenig@amd.com" moz-do-not-send="true"><Christian.Koenig@amd.com></a>;
Kuehling, Felix <a class="moz-txt-link-rfc2396E" href="mailto:Felix.Kuehling@amd.com" moz-do-not-send="true"><Felix.Kuehling@amd.com></a>;
Christian König <a class="moz-txt-link-rfc2396E" href="mailto:ckoenig.leichtzumerken@gmail.com" moz-do-not-send="true"><ckoenig.leichtzumerken@gmail.com></a>;
<a class="moz-txt-link-abbreviated
moz-txt-link-freetext" href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">amd-gfx@lists.freedesktop.org</a>;
Wentland, Harry <a class="moz-txt-link-rfc2396E" href="mailto:Harry.Wentland@amd.com" moz-do-not-send="true"><Harry.Wentland@amd.com></a><br>
<b>Cc:</b> Deucher, Alexander <a class="moz-txt-link-rfc2396E" href="mailto:Alexander.Deucher@amd.com" moz-do-not-send="true"><Alexander.Deucher@amd.com></a>;
Fan, Shikang <a class="moz-txt-link-rfc2396E" href="mailto:Shikang.Fan@amd.com" moz-do-not-send="true"><Shikang.Fan@amd.com></a><br>
<b>Subject:</b> RE: <span style="font-family:"Microsoft
YaHei",sans-serif" lang="ZH-CN"> 回复</span>:
[PATCH] drm/amdgpu: Ignore first evction failure
during suspend<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Chris,<o:p></o:p></p>
<p class="MsoNormal">I can dump these busy BOs with their
alloc/free stack later today.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">BTW, the two evictions and the kfd
suspend are all called before hw_fini. IOW, between
phase 1 and phase 2. SDMA is turned only in phase2. So
current code works fine maybe.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Koenig, Christian
<<a href="mailto:Christian.Koenig@amd.com" moz-do-not-send="true" class="moz-txt-link-freetext">Christian.Koenig@amd.com</a>>
<br>
<b>Sent:</b> Wednesday, September 13, 2023 10:29 PM<br>
<b>To:</b> Kuehling, Felix <<a href="mailto:Felix.Kuehling@amd.com" moz-do-not-send="true" class="moz-txt-link-freetext">Felix.Kuehling@amd.com</a>>;
Christian König <<a href="mailto:ckoenig.leichtzumerken@gmail.com" moz-do-not-send="true" class="moz-txt-link-freetext">ckoenig.leichtzumerken@gmail.com</a>>;
Pan, Xinhui <<a href="mailto:Xinhui.Pan@amd.com" moz-do-not-send="true" class="moz-txt-link-freetext">Xinhui.Pan@amd.com</a>>;
<a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true" class="moz-txt-link-freetext">amd-gfx@lists.freedesktop.org</a>;
Wentland, Harry <<a href="mailto:Harry.Wentland@amd.com" moz-do-not-send="true" class="moz-txt-link-freetext">Harry.Wentland@amd.com</a>><br>
<b>Cc:</b> Deucher, Alexander <<a href="mailto:Alexander.Deucher@amd.com" moz-do-not-send="true" class="moz-txt-link-freetext">Alexander.Deucher@amd.com</a>>;
Fan, Shikang <<a href="mailto:Shikang.Fan@amd.com" moz-do-not-send="true" class="moz-txt-link-freetext">Shikang.Fan@amd.com</a>><br>
<b>Subject:</b> Re: <span style="font-family:"Microsoft
YaHei",sans-serif" lang="ZH-CN"> 回复</span>:
[PATCH] drm/amdgpu: Ignore first evction failure
during suspend<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">[+Harry]<o:p></o:p></p>
<div>
<p class="MsoNormal">Am 13.09.23 um 15:54 schrieb Felix
Kuehling:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On 2023-09-13 4:07, Christian
König wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">[+Fleix]<br>
<br>
Well that looks like quite a serious bug.<br>
<br>
If I'm not completely mistaken the KFD work item
tries to restore the process by moving BOs into
memory even after the suspend freeze. Normally work
items are frozen together with the user space
processes unless explicitly marked as not freezable.<br>
<br>
That this causes problem during the first eviction
phase is just the tip of the iceberg here. If a BO
is moved into invisible memory during this we
wouldn't be able to get it out of that in the second
phase because SDMA and hw is already turned off.<br>
<br>
@Felix any idea how that can happen? Have you guys
marked a work item / work queue as not freezable?<o:p></o:p></p>
</blockquote>
<p>We don't set anything to non-freezable in KFD.<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>Regards,<br>
Felix<o:p></o:p></p>
<p><o:p> </o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Or maybe the display guys?<o:p></o:p></p>
</blockquote>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
Do you guys in the display do any delayed update in a
work item which is marked as not-freezable?<br>
<br>
Otherwise I have absolutely no idea what's going on
here.<br>
<br>
Thanks,<br>
Christian.<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
@Xinhui please investigate what work item that is
and where that is coming from. Something like "if
(adev->in_suspend) dump_stack();" in the right
place should probably do it.<br>
<br>
Thanks,<br>
Christian.<o:p></o:p></p>
<div>
<p class="MsoNormal">Am 13.09.23 um 07:13 schrieb
Pan, Xinhui:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p style="margin:5.0pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:blue">[AMD
Official Use Only - General]<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">I
notice that only user space process are
frozen on my side. kthread and workqueue
keeps running. Maybe some kernel configs are
not enabled.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">I
made one module which just prints something
like i++ with mutex lock both in workqueue
and kthread. I paste some logs below.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438619.696196]
XH: 14 from workqueue <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438619.700193]
XH: 15 from kthread<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.394335]
PM: suspend entry (deep)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.399619]
Filesystems sync: 0.001 seconds<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.403887]
PM: Preparing system for sleep (deep)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.409299]
Freezing user space processes<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.414862]
Freezing user space processes completed
(elapsed 0.001 seconds)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.421881]
OOM killer disabled.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.425197]
Freezing remaining freezable tasks<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.430890]
Freezing remaining freezable tasks
completed (elapsed 0.001 seconds)<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438620.438348]
PM: Suspending system (deep)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">.....<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438623.746038]
PM: suspend of devices complete after
3303.137 msecs <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438623.752125]
PM: start suspend of devices complete
after 3309.713 msecs<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438623.758722]
PM: suspend debug: Waiting for 5
second(s).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438623.792166]
XH: 22 from kthread<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">[438623.824140]
XH: 23 from workqueue<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">So
BOs definitely can be in use during suspend.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">Even
if kthread or workqueue can be stopped with
one special kernel config. I think suspend
can only stop the workqueue with its
callback finish. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">otherwise
something like below makes things crazy.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">LOCK
BO<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">do
something<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">
-> schedule or wait, anycode might
sleep. Stopped by suspend now? no, i think.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">UNLOCK
BO<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">I
do tests with cmds below.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">echo
devices > /sys/power/pm_test <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">echo
0 > /sys/power/pm_async<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">echo
1 > /sys/power/pm_print_times<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">echo
1 > /sys/power/pm_debug_messages<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">echo
1 >
/sys/module/amdgpu/parameters/debug_evictions<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">./kfd.sh
--gtest_filter=KFDEvictTest.BasicTest<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">pm-suspend<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">thanks<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black">xinhui<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Aptos",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" style="text-align:center" align="center">
<hr width="98%" size="1" align="center"> </div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="font-family:"Microsoft
YaHei",sans-serif;color:black" lang="ZH-CN">发件人</span><span style="color:black">:</span></b><span style="color:black"> Christian König <a href="mailto:ckoenig.leichtzumerken@gmail.com" moz-do-not-send="true"><ckoenig.leichtzumerken@gmail.com></a><br>
</span><b><span style="font-family:"Microsoft
YaHei",sans-serif;color:black" lang="ZH-CN">发送时间</span><span style="color:black">:</span></b><span style="color:black"> 2023</span><span style="font-family:"Microsoft
YaHei",sans-serif;color:black" lang="ZH-CN">年</span><span style="color:black">9</span><span style="font-family:"Microsoft
YaHei",sans-serif;color:black" lang="ZH-CN">月</span><span style="color:black">12</span><span style="font-family:"Microsoft
YaHei",sans-serif;color:black" lang="ZH-CN">日</span><span style="color:black"> 17:01<br>
</span><b><span style="font-family:"Microsoft
YaHei",sans-serif;color:black" lang="ZH-CN">收件人</span><span style="color:black">:</span></b><span style="color:black"> Pan, Xinhui <a href="mailto:Xinhui.Pan@amd.com" moz-do-not-send="true"><Xinhui.Pan@amd.com></a>;
<a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true" class="moz-txt-link-freetext">
amd-gfx@lists.freedesktop.org</a> <a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">
<amd-gfx@lists.freedesktop.org></a><br>
</span><b><span style="font-family:"Microsoft
YaHei",sans-serif;color:black" lang="ZH-CN">抄送</span><span style="color:black">:</span></b><span style="color:black"> Deucher, Alexander <a href="mailto:Alexander.Deucher@amd.com" moz-do-not-send="true"><Alexander.Deucher@amd.com></a>;
Koenig, Christian <a href="mailto:Christian.Koenig@amd.com" moz-do-not-send="true"><Christian.Koenig@amd.com></a>;
Fan, Shikang <a href="mailto:Shikang.Fan@amd.com" moz-do-not-send="true"><Shikang.Fan@amd.com></a><br>
</span><b><span style="font-family:"Microsoft
YaHei",sans-serif;color:black" lang="ZH-CN">主题</span><span style="color:black">:</span></b><span style="color:black"> Re: [PATCH] drm/amdgpu:
Ignore first evction failure during suspend</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">When
amdgpu_device_suspend() is called processes
should be frozen <br>
already. In other words KFD queues etc...
should already be idle.<br>
<br>
So when the eviction fails here we missed
something previously and that <br>
in turn can cause tons amount of problems.<br>
<br>
So ignoring those errors is most likely not
a good idea at all.<br>
<br>
Regards,<br>
Christian.<br>
<br>
Am 12.09.23 um 02:21 schrieb Pan, Xinhui:<br>
> [AMD Official Use Only - General]<br>
><br>
> Oh yep, Pinned BO is moved to other LRU
list, So eviction fails because of other
reason.<br>
> I will change the comments in the
patch.<br>
> The problem is eviction fails as many
reasons, say, BO is locked.<br>
> ASAIK, kfd will stop the queues and
flush some evict/restore work in its suspend
callback. SO the first eviction before kfd
callback likely fails.<br>
><br>
> -----Original Message-----<br>
> From: Christian König <a href="mailto:ckoenig.leichtzumerken@gmail.com" moz-do-not-send="true"><ckoenig.leichtzumerken@gmail.com></a><br>
> Sent: Friday, September 8, 2023 2:49 PM<br>
> To: Pan, Xinhui <a href="mailto:Xinhui.Pan@amd.com" moz-do-not-send="true"><Xinhui.Pan@amd.com></a>;
<a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true" class="moz-txt-link-freetext">
amd-gfx@lists.freedesktop.org</a><br>
> Cc: Deucher, Alexander <a href="mailto:Alexander.Deucher@amd.com" moz-do-not-send="true"><Alexander.Deucher@amd.com></a>;
Koenig, Christian <a href="mailto:Christian.Koenig@amd.com" moz-do-not-send="true"><Christian.Koenig@amd.com></a>;
Fan, Shikang <a href="mailto:Shikang.Fan@amd.com" moz-do-not-send="true"><Shikang.Fan@amd.com></a><br>
> Subject: Re: [PATCH] drm/amdgpu: Ignore
first evction failure during suspend<br>
><br>
> Am 08.09.23 um 05:39 schrieb xinhui
pan:<br>
>> Some BOs might be pinned. So the
first eviction's failure will abort<br>
>> the suspend sequence. These pinned
BOs will be unpined afterwards<br>
>> during suspend.<br>
> That doesn't make much sense since
pinned BOs don't cause eviction failure
here.<br>
><br>
> What exactly is the error code you see?<br>
><br>
> Christian.<br>
><br>
>> Actaully it has evicted most BOs,
so that should stil work fine in<br>
>> sriov full access mode.<br>
>><br>
>> Fixes: 47ea20762bb7 ("drm/amdgpu:
Add an extra evict_resource call<br>
>> during device_suspend.")<br>
>> Signed-off-by: xinhui pan <a href="mailto:xinhui.pan@amd.com" moz-do-not-send="true"><xinhui.pan@amd.com></a><br>
>> ---<br>
>>
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |
9 +++++----<br>
>> 1 file changed, 5 insertions(+),
4 deletions(-)<br>
>><br>
>> diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br>
>>
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br>
>> index 5c0e2b766026..39af526cdbbe
100644<br>
>> ---
a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br>
>> +++
b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br>
>> @@ -4148,10 +4148,11 @@ int
amdgpu_device_suspend(struct drm_device<br>
>> *dev, bool fbcon)<br>
>><br>
>> adev->in_suspend = true;<br>
>><br>
>> - /* Evict the majority of BOs
before grabbing the full access */<br>
>> - r =
amdgpu_device_evict_resources(adev);<br>
>> - if (r)<br>
>> - return r;<br>
>> + /* Try to evict the majority
of BOs before grabbing the full access<br>
>> + * Ignore the ret val at
first place as we will unpin some BOs if any<br>
>> + * afterwards.<br>
>> + */<br>
>> +
(void)amdgpu_device_evict_resources(adev);<br>
>><br>
>> if (amdgpu_sriov_vf(adev)) {<br>
>>
amdgpu_virt_fini_data_exchange(adev);<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
<br>
</blockquote>
</blockquote>
<br>
</body>
</html>