<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Apr 14, 2022, at 1:31 AM, Andrey Grodzovsky <<a href="mailto:andrey.grodzovsky@amd.com" class="">andrey.grodzovsky@amd.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">
<p class=""><br class="">
</p>
<div class="moz-cite-prefix">On 2022-04-13 12:03, Shuotao Xu wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:5A64FAEA-CCE8-4EB6-8E7B-852D4F384255@microsoft.com" class="">
<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Apr 11, 2022, at 11:52 PM, Andrey Grodzovsky <<a href="mailto:andrey.grodzovsky@amd.com" class="moz-txt-link-freetext" moz-do-not-send="true">andrey.grodzovsky@amd.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="caret-color: rgb(0, 0, 0);
font-family: Helvetica; font-size: 12px; font-style:
normal; font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">[Some
people who received this message don't often get email from<span class="Apple-converted-space"> </span></span><a href="mailto:andrey.grodzovsky@amd.com" style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: auto;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: auto; word-spacing: 0px;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width:
0px;" class="moz-txt-link-freetext" moz-do-not-send="true">andrey.grodzovsky@amd.com</a><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;
float: none; display: inline !important;" class="">.
Learn why this is important at<span class="Apple-converted-space"> </span></span><a href="http://aka.ms/LearnAboutSenderIdentification" style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: auto;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: auto; word-spacing: 0px;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width:
0px;" class="moz-txt-link-freetext" moz-do-not-send="true">http://aka.ms/LearnAboutSenderIdentification</a><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;
float: none; display: inline !important;" class="">.]</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">On
2022-04-08 21:28, Shuotao Xu wrote:</span><br style="caret-color: rgb(0, 0, 0);
font-family: Helvetica; font-size: 12px; font-style:
normal; font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<blockquote type="cite" style="font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px; text-transform:
none; white-space: normal; widows: auto; word-spacing:
0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
<blockquote type="cite" class="">On Apr 8, 2022, at 11:28 PM, Andrey Grodzovsky <<a href="mailto:andrey.grodzovsky@amd.com" class="moz-txt-link-freetext" moz-do-not-send="true">andrey.grodzovsky@amd.com</a>> wrote:<br class="">
<br class="">
[Some people who received this message don't often get email from <a href="mailto:andrey.grodzovsky@amd.com" class="moz-txt-link-freetext" moz-do-not-send="true">
andrey.grodzovsky@amd.com</a>. Learn why this is important at <a href="http://aka.ms/LearnAboutSenderIdentification" class="moz-txt-link-freetext" moz-do-not-send="true">
http://aka.ms/LearnAboutSenderIdentification</a>.]<br class="">
<br class="">
On 2022-04-08 04:45, Shuotao Xu wrote:<br class="">
<blockquote type="cite" class="">Adding PCIe Hotplug Support for AMDKFD: the support of hot-plug of GPU<br class="">
devices can open doors for many advanced applications in data center<br class="">
in the next few years, such as for GPU resource<br class="">
disaggregation. Current AMDKFD does not support hotplug out b/o the<br class="">
following reasons:<br class="">
<br class="">
1. During PCIe removal, decrement KFD lock which was incremented at<br class="">
the beginning of hw fini; otherwise kfd_open later is going to<br class="">
fail.<br class="">
</blockquote>
I assumed you read my comment last time, still you do same approach.<br class="">
More in details bellow<br class="">
</blockquote>
Aha, I like your fix:) I was not familiar with drm APIs so just only half understood your comment last time.<br class="">
<br class="">
BTW, I tried hot-plugging out a GPU when rocm application is still running.<br class="">
From dmesg, application is still trying to access the removed kfd device, and are met with some errors.<br class="">
</blockquote>
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">Application
us supposed to keep running, it holds the drm_device</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">reference
as long as it has an open</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">FD
to the device and final cleanup will come only after the app will die</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">thus
releasing the FD and the last</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">drm_device
reference.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<blockquote type="cite" style="font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px; text-transform:
none; white-space: normal; widows: auto; word-spacing:
0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
Application would hang and not exiting in this case.<br class="">
</blockquote>
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
</div>
</blockquote>
<div class=""><br class="">
</div>
Actually I tried kill -7 $pid, and the process exists. The dmesg has some warning though.</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">[ 711.769977] WARNING: CPU: 23 PID: 344 at .../amdgpu-rocm5.0.2/src/amd/amdgpu/amdgpu_object.c:1336 amdgpu_bo_release_notify+0x150/0x160 [amdgpu]</div>
<div class="">[ 711.770528] Modules linked in: amdgpu(OE) amdttm(OE) amd_sched(OE) amdkcl(OE) iommu_v2 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack
nf_defrag_ipv6 nf_defrag_ipv4 ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay binfmt_misc intel_rapl_msr i40iw intel_rapl_common skx_edac nfit x86_pkg_temp_thermal intel_powerclamp
coretemp kvm_intel rpcrdma kvm sunrpc ipmi_ssif ib_umad ib_ipoib rdma_ucm irqbypass rapl joydev acpi_ipmi input_leds intel_cstate ipmi_si ipmi_devintf mei_me mei intel_pch_thermal ipmi_msghandler ioatdma mac_hid lpc_ich dca acpi_power_meter acpi_pad sch_fq_codel
ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi pci_stub ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor</div>
<div class="">[ 711.779359] raid6_pq libcrc32c raid1 raid0 multipath linear mlx5_ib ib_uverbs ib_core ast drm_vram_helper i2c_algo_bit drm_ttm_helper ttm drm_kms_helper syscopyarea crct10dif_pclmul crc32_pclmul ghash_clmulni_intel sysfillrect uas hid_generic
sysimgblt aesni_intel mlx5_core fb_sys_fops crypto_simd usbhid cryptd drm i40e pci_hyperv_intf usb_storage glue_helper mlxfw hid ahci libahci wmi</div>
<div class="">[ 711.779752] CPU: 23 PID: 344 Comm: kworker/23:1 Tainted: G W OE 5.11.0+ #1</div>
<div class="">[ 711.779755] Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 2.1 08/14/2018</div>
<div class="">[ 711.779756] Workqueue: kfd_process_wq kfd_process_wq_release [amdgpu]</div>
<div class="">[ 711.779955] RIP: 0010:amdgpu_bo_release_notify+0x150/0x160 [amdgpu]</div>
<div class="">[ 711.780141] Code: e8 b5 af 34 f4 e9 1f ff ff ff 48 39 c2 74 07 0f 0b e9 69 ff ff ff 4c 89 e7 e8 3c b4 16 00 e9 5c ff ff ff e8 a2 ce fd f3 eb cf <0f> 0b eb cb e8 d7 02 34 f4 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55</div>
<div class="">[ 711.780143] RSP: 0018:ffffa8100dd67c30 EFLAGS: 00010282</div>
<div class="">[ 711.780145] RAX: 00000000ffffffea RBX: ffff89980e792058 RCX: 0000000000000000</div>
<div class="">[ 711.780147] RDX: 0000000000000000 RSI: ffff89a8f9ad8870 RDI: ffff89a8f9ad8870</div>
<div class="">[ 711.780148] RBP: ffffa8100dd67c50 R08: 0000000000000000 R09: fffffffffff99b18</div>
<div class="">[ 711.780149] R10: ffffa8100dd67bd0 R11: ffffa8100dd67908 R12: ffff89980e792000</div>
<div class="">[ 711.780151] R13: ffff89980e792058 R14: ffff89980e7921bc R15: dead000000000100</div>
<div class="">[ 711.780152] FS: 0000000000000000(0000) GS:ffff89a8f9ac0000(0000) knlGS:0000000000000000</div>
<div class="">[ 711.780154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033</div>
<div class="">[ 711.780156] CR2: 00007ffddac6f71f CR3: 00000030bb80a003 CR4: 00000000007706e0</div>
<div class="">[ 711.780157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000</div>
<div class="">[ 711.780159] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400</div>
<div class="">[ 711.780160] PKRU: 55555554</div>
<div class="">[ 711.780161] Call Trace:</div>
<div class="">[ 711.780163] ttm_bo_release+0x2ae/0x320 [amdttm]</div>
<div class="">[ 711.780169] amdttm_bo_put+0x30/0x40 [amdttm]</div>
<div class="">[ 711.780357] amdgpu_bo_unref+0x1e/0x30 [amdgpu]</div>
<div class="">[ 711.780543] amdgpu_gem_object_free+0x8b/0x160 [amdgpu]</div>
<div class="">[ 711.781119] drm_gem_object_free+0x1d/0x30 [drm]</div>
<div class="">[ 711.781489] amdgpu_amdkfd_gpuvm_free_memory_of_gpu+0x34a/0x380 [amdgpu]</div>
<div class="">[ 711.782044] kfd_process_device_free_bos+0xe0/0x130 [amdgpu]</div>
<div class="">[ 711.782611] kfd_process_wq_release+0x286/0x380 [amdgpu]</div>
<div class="">[ 711.783172] process_one_work+0x236/0x420</div>
<div class="">[ 711.783543] worker_thread+0x34/0x400</div>
<div class="">[ 711.783911] ? process_one_work+0x420/0x420</div>
<div class="">[ 711.784279] kthread+0x126/0x140</div>
<div class="">[ 711.784653] ? kthread_park+0x90/0x90</div>
<div class="">[ 711.785018] ret_from_fork+0x22/0x30</div>
<div class="">[ 711.785387] ---[ end trace d8f50f6594817c84 ]---</div>
<div class="">[ 711.798716] [drm] amdgpu: ttm finalized</div>
</div>
</blockquote>
<p class=""><br class="">
</p>
<p class="">So it means the process was stuck in some wait_event_killable (maybe here drm_sched_entity_flush) - you can try 'cat/proc/$process_pid/stack' maybe before<br class="">
you kill it to see where it was stuck so we can go from there.<br class="">
</p>
<p class=""><br class="">
</p>
<blockquote type="cite" cite="mid:5A64FAEA-CCE8-4EB6-8E7B-852D4F384255@microsoft.com" class="">
<div class="">
<div class=""><br class="">
</div>
</div>
<div class="">
<blockquote type="cite" class="">
<div class=""><br style="caret-color: rgb(0, 0, 0);
font-family: Helvetica; font-size: 12px; font-style:
normal; font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">For
graphic apps what i usually see is a crash because of sigsev when</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">the
app tries to access</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">an
unmapped MMIO region on the device. I haven't tested for compute</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">stack
and so there might</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">be
something I haven't covered. Hang could mean for example waiting on a</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">fence
which is not being</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">signaled
- please provide full dmesg from this case.</span><br style="caret-color: rgb(0, 0,
0); font-family: Helvetica; font-size: 12px; font-style:
normal; font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<blockquote type="cite" style="font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px; text-transform:
none; white-space: normal; widows: auto; word-spacing:
0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
Do you have any good suggestions on how to fix it down the line? (HIP runtime/libhsakmt or driver)<br class="">
<br class="">
[64036.631333] amdgpu: amdgpu_vm_bo_update failed<br class="">
[64036.631702] amdgpu: validate_invalid_user_pages: update PTE failed<br class="">
[64036.640754] amdgpu: amdgpu_vm_bo_update failed<br class="">
[64036.641120] amdgpu: validate_invalid_user_pages: update PTE failed<br class="">
[64036.650394] amdgpu: amdgpu_vm_bo_update failed<br class="">
[64036.650765] amdgpu: validate_invalid_user_pages: update PTE failed<br class="">
</blockquote>
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
</div>
</blockquote>
<div class=""><br class="">
</div>
The full dmesg will just the repetition of those two messages,</div>
<div class="">
<div class="">[186885.764079] amdgpu 0000:43:00.0: amdgpu: amdgpu: finishing device.</div>
<div class="">[186885.766916] [drm] free PSP TMR buffer</div>
<div class="">[186893.868173] amdgpu: amdgpu_vm_bo_update failed</div>
<div class="">[186893.868235] amdgpu: validate_invalid_user_pages: update PTE failed</div>
<div class="">[186893.876154] amdgpu: amdgpu_vm_bo_update failed</div>
<div class="">[186893.876190] amdgpu: validate_invalid_user_pages: update PTE failed</div>
<div class="">[186893.884150] amdgpu: amdgpu_vm_bo_update failed</div>
<div class="">[186893.884185] amdgpu: validate_invalid_user_pages: update PTE failed</div>
</div>
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class="">
<div class=""><br style="caret-color: rgb(0, 0, 0);
font-family: Helvetica; font-size: 12px; font-style:
normal; font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">This
just probably means trying to update PTEs after the physical device</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">is
gone - we usually avoid this by</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">first
trying to do all HW shutdowns early before PCI remove completion</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">but
when it's really tricky by</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">protecting
HW access sections with drm_dev_enter/exit scope.</span><br style="caret-color:
rgb(0, 0, 0); font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">For
this particular error it would be the best to flush</span><br style="caret-color: rgb(0,
0, 0); font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">info->restore_userptr_work
before the end of</span><br style="caret-color: rgb(0, 0, 0);
font-family: Helvetica; font-size: 12px; font-style:
normal; font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">amdgpu_pci_remove
(rejecting new process creation and calling</span><br style="caret-color:
rgb(0, 0, 0); font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">cancel_delayed_work_sync(&process_info->restore_userptr_work)
for all</span><br style="caret-color: rgb(0, 0, 0);
font-family: Helvetica; font-size: 12px; font-style:
normal; font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">running
processes)</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">somewhere
in amdgpu_pci_remove.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
</div>
</blockquote>
I tried something like *kfd_process_ref_release* which I think did what you described, but it did not work.</div>
</blockquote>
<p class=""><br class="">
</p>
<p class="">I don't see how kfd_process_ref_release is the same as I mentioned above, what i meant is calling the code above within kgd2kfd_suspend (where you tried to call kfd_kill_all_user_processes bellow)
<br class="">
</p>
</div>
</div>
</blockquote>
Yes, you are right. It was not called by it. <br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">
<p class=""></p>
<p class=""><br class="">
</p>
<blockquote type="cite" cite="mid:5A64FAEA-CCE8-4EB6-8E7B-852D4F384255@microsoft.com" class="">
<div class=""><br class="">
</div>
<div class="">Instead I tried to kill the process from the kernel, but the amdgpu could **only** be hot-plugged in back successfully only if there was no rocm kernel running when it was plugged out. If not, amdgpu_probe will just hang later. (Maybe because
amdgpu was plugged out while running state, it leaves a bad HW state which causes probe to hang).</div>
</blockquote>
<p class=""><br class="">
</p>
<p class="">We usually do asic_reset during probe to reset all HW state (checlk if amdgpu_device_init->amdgpu_asic_reset is running when you plug back).
<br class="">
</p>
</div>
</div>
</blockquote>
OK<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">
<p class=""></p>
<p class=""> <br class="">
</p>
<blockquote type="cite" cite="mid:5A64FAEA-CCE8-4EB6-8E7B-852D4F384255@microsoft.com" class="">
<div class=""><br class="">
</div>
<div class="">I don’t know if this is a viable solution worth pursuing, but I attached the diff anyway.</div>
<div class=""><br class="">
</div>
<div class="">Another solution could be let compute stack user mode detect a topology change via <span class="">generation_count change, and abort gracefully there.</span></div>
<div class=""><br class="">
</div>
<div class="">diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c</div>
<div class="">index 4e7d9cb09a69..79b4c9b84cd0 100644</div>
<div class="">--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c</div>
<div class="">+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c</div>
<div class="">@@ -697,12 +697,15 @@ void kgd2kfd_suspend(struct kfd_dev *kfd, bool run_pm, bool force)</div>
<div class=""> return;</div>
<div class=""><br class="">
</div>
<div class=""> /* for runtime suspend, skip locking kfd */</div>
<div class="">- if (!run_pm) {</div>
<div class="">+ if (!run_pm && !drm_dev_is_unplugged(kfd->ddev)) {</div>
<div class=""> /* For first KFD device suspend all the KFD processes */</div>
<div class=""> if (atomic_inc_return(&kfd_locked) == 1)</div>
<div class=""> kfd_suspend_all_processes(force);</div>
<div class=""> }</div>
<div class=""><br class="">
</div>
<div class="">+ if (drm_dev_is_unplugged(kfd->ddev))</div>
<div class="">+ kfd_kill_all_user_processes();</div>
<div class="">+</div>
<div class=""> kfd->dqm->ops.stop(kfd->dqm);</div>
<div class=""> kfd_iommu_suspend(kfd);</div>
<div class=""> }</div>
<div class="">diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h</div>
<div class="">index 55c9e1922714..84cbcd857856 100644</div>
<div class="">--- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h</div>
<div class="">+++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h</div>
<div class="">@@ -1065,6 +1065,7 @@ void kfd_unref_process(struct kfd_process *p);</div>
<div class=""> int kfd_process_evict_queues(struct kfd_process *p, bool force);</div>
<div class=""> int kfd_process_restore_queues(struct kfd_process *p);</div>
<div class=""> void kfd_suspend_all_processes(bool force);</div>
<div class="">+void kfd_kill_all_user_processes(void);</div>
<div class=""> /*</div>
<div class=""> * kfd_resume_all_processes:</div>
<div class=""> * bool sync: If kfd_resume_all_processes() should wait for the</div>
<div class="">diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c</div>
<div class="">index 6cdc855abb6d..fb0c753b682c 100644</div>
<div class="">--- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c</div>
<div class="">+++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c</div>
<div class="">@@ -2206,6 +2206,24 @@ void kfd_suspend_all_processes(bool force)</div>
<div class=""> srcu_read_unlock(&kfd_processes_srcu, idx);</div>
<div class=""> }</div>
<div class=""><br class="">
</div>
<div class="">+</div>
<div class="">+void kfd_kill_all_user_processes(void)</div>
<div class="">+{</div>
<div class="">+ struct kfd_process *p;</div>
<div class="">+ struct amdkfd_process_info *p_info;</div>
<div class="">+ unsigned int temp;</div>
<div class="">+ int idx = srcu_read_lock(&kfd_processes_srcu);</div>
<div class="">+</div>
<div class="">+ pr_info("Killing all processes\n");</div>
<div class="">+ hash_for_each_rcu(kfd_processes_table, temp, p, kfd_processes) {</div>
<div class="">+ p_info = p->kgd_process_info;</div>
<div class="">+ pr_info("Killing processes, pid = %d", pid_nr(p_info->pid));</div>
<div class="">+ kill_pid(p_info->pid, SIGBUS, 1);</div>
</blockquote>
<p class=""><br class="">
</p>
<p class="">From looking into kill_pid I see it only sends a signal but doesn't wait for completion, it would make sense to wait for completion here. In any case I would actually try to put here<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline !important;" class=""><br class="">
</span></p>
</div>
</div>
</blockquote>
I have made a version which does that with some atomic counters. Please read later in the diff.<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">
<p class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline !important;" class=""></span><span style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; text-align: start; text-indent: 0px; text-transform:
none; white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none; float:
none; display: inline !important;" class=""><font size="4" class=""><br class="">
</font></span></p>
<p class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline !important;" class=""><font size="4" class="">hash_for_each_rcu(</font></span><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline !important;" class=""><font size="4" class="">p_info)
<br class="">
cancel_delayed_work_sync(&</font></span><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline !important;" class=""><font size="4" class=""><span style="caret-color: rgb(0, 0, 0);
font-family: Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; text-align: start; text-indent: 0px; text-transform:
none; white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;
float: none; display: inline !important;" class=""></span><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;
float: none; display: inline !important;" class=""><font size="4" class="">p_info</font></span>->restore_userptr_work)
<br class="">
</font></span></p>
<p class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline !important;" class=""><font size="4" class="">instead
at least that what i meant in the previous mail. </font></span></p>
</div>
</div>
</blockquote>
<div>I actually tried that earlier, and it did not work. Application still keeps running, and you have to send a kill to the user process.</div>
<div><br class="">
</div>
<div>I have made the following version. It waits for processes to terminate synchronously after sending SIGBUS. After that it does the real work of amdgpu_pci_remove.</div>
<div>However, it hangs at amdgpu_device_ip_fini_early when it is trying to deinit ip_block 6 <sdma_v4_0> (<a href="https://gitlab.freedesktop.org/agd5f/linux/-/blob/amd-staging-drm-next/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c#L2818" class="">https://gitlab.freedesktop.org/agd5f/linux/-/blob/amd-staging-drm-next/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c#L2818</a>).
I assume that there are still some inflight dma, therefore fini of this ip block thus hangs? </div>
<div><br class="">
</div>
<div>The following is an excerpt of the dmesg: please excuse for putting my own pr_info, but I hope you get my point of where it hangs.</div>
<div><br class="">
</div>
<div>
<div>[ 392.344735] amdgpu: all processes has been fully released</div>
<div>[ 392.346557] amdgpu: amdgpu_acpi_fini done</div>
<div>[ 392.346568] amdgpu 0000:b3:00.0: amdgpu: amdgpu: finishing device.</div>
<div>[ 392.349238] amdgpu: amdgpu_device_ip_fini_early enter ip_blocks = 9</div>
<div>[ 392.349248] amdgpu: Free mem_obj = 000000007bf54275, range_start = 14, range_end = 14</div>
<div>[ 392.350299] amdgpu: Free mem_obj = 00000000a85bc878, range_start = 12, range_end = 12</div>
<div>[ 392.350304] amdgpu: Free mem_obj = 00000000b8019e32, range_start = 13, range_end = 13</div>
<div>[ 392.350308] amdgpu: Free mem_obj = 000000002d296168, range_start = 4, range_end = 11</div>
<div>[ 392.350313] amdgpu: Free mem_obj = 000000001fc4f934, range_start = 0, range_end = 3</div>
<div>[ 392.350322] amdgpu: amdgpu_amdkfd_suspend(adev, false) done</div>
<div>[ 392.350672] amdgpu: hw_fini of IP block[8] <jpeg_v2_5> done 0</div>
<div>[ 392.350679] amdgpu: hw_fini of IP block[7] <vcn_v2_5> done 0</div>
</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div>
<div>diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c</div>
<div>index 8fa9b86ac9d2..c0b27f722281 100644</div>
<div>--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c</div>
<div>+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c</div>
<div>@@ -188,6 +188,12 @@ void amdgpu_amdkfd_interrupt(struct amdgpu_device *adev,</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>kgd2kfd_interrupt(adev->kfd.dev, ih_ring_entry);</div>
<div> }</div>
<div> </div>
<div>+void amdgpu_amdkfd_kill_all_processes(struct amdgpu_device *adev)</div>
<div>+{</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>if (adev->kfd.dev)</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>kgd2kfd_kill_all_user_processes(adev->kfd.dev);</div>
<div>+}</div>
<div>+</div>
<div> void amdgpu_amdkfd_suspend(struct amdgpu_device *adev, bool run_pm)</div>
<div> {</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>if (adev->kfd.dev)</div>
<div>diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h</div>
<div>index 27c74fcec455..f4e485d60442 100644</div>
<div>--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h</div>
<div>+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h</div>
<div>@@ -141,6 +141,7 @@ struct amdkfd_process_info {</div>
<div> int amdgpu_amdkfd_init(void);</div>
<div> void amdgpu_amdkfd_fini(void);</div>
<div> </div>
<div>+void amdgpu_amdkfd_kill_all_processes(struct amdgpu_device *adev);</div>
<div> void amdgpu_amdkfd_suspend(struct amdgpu_device *adev, bool run_pm);</div>
<div> int amdgpu_amdkfd_resume_iommu(struct amdgpu_device *adev);</div>
<div> int amdgpu_amdkfd_resume(struct amdgpu_device *adev, bool run_pm, bool sync);</div>
<div>@@ -405,6 +406,7 @@ bool kgd2kfd_device_init(struct kfd_dev *kfd,</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>const struct kgd2kfd_shared_resources *gpu_resources);</div>
<div> void kgd2kfd_device_exit(struct kfd_dev *kfd);</div>
<div> void kgd2kfd_suspend(struct kfd_dev *kfd, bool run_pm, bool force);</div>
<div>+void kgd2kfd_kill_all_user_processes(struct kfd_dev *kfd);</div>
<div> int kgd2kfd_resume_iommu(struct kfd_dev *kfd);</div>
<div> int kgd2kfd_resume(struct kfd_dev *kfd, bool run_pm, bool sync);</div>
<div> int kgd2kfd_pre_reset(struct kfd_dev *kfd);</div>
<div>@@ -443,6 +445,9 @@ static inline void kgd2kfd_suspend(struct kfd_dev *kfd, bool run_pm, bool force)</div>
<div> {</div>
<div> }</div>
<div> </div>
<div>+void kgd2kfd_kill_all_user_processes(struct kfd_dev *kfd){</div>
<div>+}</div>
<div>+</div>
<div> static int __maybe_unused kgd2kfd_resume_iommu(struct kfd_dev *kfd)</div>
<div> {</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>return 0;</div>
<div>diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c</div>
<div>index 3d5fc0751829..af6fe5080cfa 100644</div>
<div>--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c</div>
<div>+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c</div>
<div>@@ -2101,6 +2101,9 @@ amdgpu_pci_remove(struct pci_dev *pdev)</div>
<div> {</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>struct drm_device *dev = pci_get_drvdata(pdev);</div>
<div> </div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>/* kill all kfd processes before drm_dev_unplug */</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>amdgpu_amdkfd_kill_all_processes(drm_to_adev(dev));</div>
<div>+</div>
<div> #ifdef HAVE_DRM_DEV_UNPLUG</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>drm_dev_unplug(dev);</div>
<div> #else</div>
<div>diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c</div>
<div>index 5504a18b5a45..480c23bef5e2 100644</div>
<div>--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c</div>
<div>+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c</div>
<div>@@ -691,6 +691,12 @@ bool kfd_is_locked(void)</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>return (atomic_read(&kfd_locked) > 0);</div>
<div> }</div>
<div> </div>
<div>+inline void kgd2kfd_kill_all_user_processes(struct kfd_dev* dev)</div>
<div>+{</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>kfd_kill_all_user_processes();</div>
<div>+}</div>
<div>+</div>
<div>+</div>
<div> void kgd2kfd_suspend(struct kfd_dev *kfd, bool run_pm, bool force)</div>
<div> {</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>if (!kfd->init_complete)</div>
<div>diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h</div>
<div>index 55c9e1922714..a35a2cb5bb9f 100644</div>
<div>--- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h</div>
<div>+++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h</div>
<div>@@ -1064,6 +1064,7 @@ static inline struct kfd_process_device *kfd_process_device_from_gpuidx(</div>
<div> void kfd_unref_process(struct kfd_process *p);</div>
<div> int kfd_process_evict_queues(struct kfd_process *p, bool force);</div>
<div> int kfd_process_restore_queues(struct kfd_process *p);</div>
<div>+void kfd_kill_all_user_processes(void);</div>
<div> void kfd_suspend_all_processes(bool force);</div>
<div> /*</div>
<div> * kfd_resume_all_processes:</div>
<div>diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c</div>
<div>index 6cdc855abb6d..17e769e6951d 100644</div>
<div>--- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c</div>
<div>+++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c</div>
<div>@@ -46,6 +46,9 @@ struct mm_struct;</div>
<div> #include "kfd_trace.h"</div>
<div> #include "kfd_debug.h"</div>
<div> </div>
<div>+static atomic_t kfd_process_locked = ATOMIC_INIT(0);</div>
<div>+static atomic_t kfd_inflight_kills = ATOMIC_INIT(0);</div>
<div>+</div>
<div> /*</div>
<div> * List of struct kfd_process (field kfd_process).</div>
<div> * Unique/indexed by mm_struct*</div>
<div>@@ -802,6 +805,9 @@ struct kfd_process *kfd_create_process(struct task_struct *thread)</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>struct kfd_process *process;</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>int ret;</div>
<div> </div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>if ( atomic_read(&kfd_process_locked) > 0 )</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>return ERR_PTR(-EINVAL);</div>
<div>+</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>if (!(thread->mm && mmget_not_zero(thread->mm)))</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>return ERR_PTR(-EINVAL);</div>
<div> </div>
<div>@@ -1126,6 +1132,10 @@ static void kfd_process_wq_release(struct work_struct *work)</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>put_task_struct(p->lead_thread);</div>
<div> </div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>kfree(p);</div>
<div>+</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>if ( atomic_read(&kfd_process_locked) > 0 ){</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>atomic_dec(&kfd_inflight_kills);</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>}</div>
<div> }</div>
<div> </div>
<div> static void kfd_process_ref_release(struct kref *ref)</div>
<div>@@ -2186,6 +2196,35 @@ static void restore_process_worker(struct work_struct *work)</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>pr_err("Failed to restore queues of pasid 0x%x\n", p->pasid);</div>
<div> }</div>
<div> </div>
<div>+void kfd_kill_all_user_processes(void)</div>
<div>+{</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>struct kfd_process *p;</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>/* struct amdkfd_process_info *p_info; */</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>unsigned int temp;</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>int idx;</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>atomic_inc(&kfd_process_locked);</div>
<div>+</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>idx = srcu_read_lock(&kfd_processes_srcu);</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>pr_info("Killing all processes\n");</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>hash_for_each_rcu(kfd_processes_table, temp, p, kfd_processes) {</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>dev_warn(kfd_device,</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>"Sending SIGBUS to process %d (pasid 0x%x)",</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>p->lead_thread->pid, p->pasid);</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>send_sig(SIGBUS, p->lead_thread, 0);</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>atomic_inc(&kfd_inflight_kills);</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>}</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>srcu_read_unlock(&kfd_processes_srcu, idx);</div>
<div>+</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>while ( atomic_read(&kfd_inflight_kills) > 0 ){</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>dev_warn(kfd_device,</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>"kfd_processes_table is not empty, going to sleep for 10ms\n");</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>msleep(10);</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>}</div>
<div>+</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>atomic_dec(&kfd_process_locked);</div>
<div>+<span class="Apple-tab-span" style="white-space:pre"> </span>pr_info("all processes has been fully released");</div>
<div>+}</div>
<div>+</div>
<div> void kfd_suspend_all_processes(bool force)</div>
<div> {</div>
<div> <span class="Apple-tab-span" style="white-space:pre"> </span>struct kfd_process *p;</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<div><br class="">
</div>
<div>Regards,</div>
<div>Shuotao</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">
<p class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline !important;" class=""><font size="4" class=""></font></span></p>
<p class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; text-align: start;
text-indent: 0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline !important;" class=""><font size="4" class="">Andrey</font><br class="">
<br class="">
</span></p>
<blockquote type="cite" cite="mid:5A64FAEA-CCE8-4EB6-8E7B-852D4F384255@microsoft.com" class="">
<div class="">+ }</div>
<div class="">+ srcu_read_unlock(&kfd_processes_srcu, idx);</div>
<div class="">+}</div>
<div class="">+</div>
<div class="">+</div>
<div class=""> int kfd_resume_all_processes(bool sync)</div>
<div class=""> {</div>
<div class=""> struct kfd_process *p;</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class=""><span style="caret-color: rgb(0, 0, 0);
font-family: Helvetica; font-size: 12px; font-style:
normal; font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">Andrey</span><br style="caret-color:
rgb(0, 0, 0); font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<br style="caret-color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant-caps: normal; font-weight: 400;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none;" class="">
<blockquote type="cite" style="font-family: Helvetica;
font-size: 12px; font-style: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px; text-transform:
none; white-space: normal; widows: auto; word-spacing:
0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
Really appreciate your help!<br class="">
<br class="">
Best,<br class="">
Shuotao<br class="">
<br class="">
<blockquote type="cite" class="">
<blockquote type="cite" class="">2. Remove redudant p2p/io links in sysfs when device is hotplugged<br class="">
out.<br class="">
<br class="">
3. New kfd node_id is not properly assigned after a new device is<br class="">
added after a gpu is hotplugged out in a system. libhsakmt will<br class="">
find this anomaly, (i.e. node_from != <dev node id> in iolinks),<br class="">
when taking a topology_snapshot, thus returns fault to the rocm<br class="">
stack.<br class="">
<br class="">
-- This patch fixes issue 1; another patch by Mukul fixes issues 2&3.<br class="">
-- Tested on a 4-GPU MI100 gpu nodes with kernel 5.13.0-kfd; kernel<br class="">
5.16.0-kfd is unstable out of box for MI100.<br class="">
---<br class="">
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 5 +++++<br class="">
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 7 +++++++<br class="">
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 +<br class="">
drivers/gpu/drm/amd/amdkfd/kfd_device.c | 13 +++++++++++++<br class="">
4 files changed, 26 insertions(+)<br class="">
<br class="">
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c<br class="">
index c18c4be1e4ac..d50011bdb5c4 100644<br class="">
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c<br class="">
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c<br class="">
@@ -213,6 +213,11 @@ int amdgpu_amdkfd_resume(struct amdgpu_device *adev, bool run_pm)<br class="">
return r;<br class="">
}<br class="">
<br class="">
+int amdgpu_amdkfd_resume_processes(void)<br class="">
+{<br class="">
+ return kgd2kfd_resume_processes();<br class="">
+}<br class="">
+<br class="">
int amdgpu_amdkfd_pre_reset(struct amdgpu_device *adev)<br class="">
{<br class="">
int r = 0;<br class="">
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h<br class="">
index f8b9f27adcf5..803306e011c3 100644<br class="">
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h<br class="">
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h<br class="">
@@ -140,6 +140,7 @@ void amdgpu_amdkfd_fini(void);<br class="">
void amdgpu_amdkfd_suspend(struct amdgpu_device *adev, bool run_pm);<br class="">
int amdgpu_amdkfd_resume_iommu(struct amdgpu_device *adev);<br class="">
int amdgpu_amdkfd_resume(struct amdgpu_device *adev, bool run_pm);<br class="">
+int amdgpu_amdkfd_resume_processes(void);<br class="">
void amdgpu_amdkfd_interrupt(struct amdgpu_device *adev,<br class="">
const void *ih_ring_entry);<br class="">
void amdgpu_amdkfd_device_probe(struct amdgpu_device *adev);<br class="">
@@ -347,6 +348,7 @@ void kgd2kfd_device_exit(struct kfd_dev *kfd);<br class="">
void kgd2kfd_suspend(struct kfd_dev *kfd, bool run_pm);<br class="">
int kgd2kfd_resume_iommu(struct kfd_dev *kfd);<br class="">
int kgd2kfd_resume(struct kfd_dev *kfd, bool run_pm);<br class="">
+int kgd2kfd_resume_processes(void);<br class="">
int kgd2kfd_pre_reset(struct kfd_dev *kfd);<br class="">
int kgd2kfd_post_reset(struct kfd_dev *kfd);<br class="">
void kgd2kfd_interrupt(struct kfd_dev *kfd, const void *ih_ring_entry);<br class="">
@@ -393,6 +395,11 @@ static inline int kgd2kfd_resume(struct kfd_dev *kfd, bool run_pm)<br class="">
return 0;<br class="">
}<br class="">
<br class="">
+static inline int kgd2kfd_resume_processes(void)<br class="">
+{<br class="">
+ return 0;<br class="">
+}<br class="">
+<br class="">
static inline int kgd2kfd_pre_reset(struct kfd_dev *kfd)<br class="">
{<br class="">
return 0;<br class="">
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br class="">
index fa4a9f13c922..5827b65b7489 100644<br class="">
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br class="">
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br class="">
@@ -4004,6 +4004,7 @@ void amdgpu_device_fini_hw(struct amdgpu_device *adev)<br class="">
if (drm_dev_is_unplugged(adev_to_drm(adev)))<br class="">
amdgpu_device_unmap_mmio(adev);<br class="">
<br class="">
+ amdgpu_amdkfd_resume_processes();<br class="">
}<br class="">
<br class="">
void amdgpu_device_fini_sw(struct amdgpu_device *adev)<br class="">
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c<br class="">
index 62aa6c9d5123..ef05aae9255e 100644<br class="">
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c<br class="">
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c<br class="">
@@ -714,6 +714,19 @@ int kgd2kfd_resume(struct kfd_dev *kfd, bool run_pm)<br class="">
return ret;<br class="">
}<br class="">
<br class="">
+/* for non-runtime resume only */<br class="">
+int kgd2kfd_resume_processes(void)<br class="">
+{<br class="">
+ int count;<br class="">
+<br class="">
+ count = atomic_dec_return(&kfd_locked);<br class="">
+ WARN_ONCE(count < 0, "KFD suspend / resume ref. error");<br class="">
+ if (count == 0)<br class="">
+ return kfd_resume_all_processes();<br class="">
+<br class="">
+ return 0;<br class="">
+}<br class="">
</blockquote>
<br class="">
It doesn't make sense to me to just increment kfd_locked in<br class="">
kgd2kfd_suspend to only decrement it again a few functions down the<br class="">
road.<br class="">
<br class="">
I suggest this instead - you only incrmemnt if not during PCI remove<br class="">
<br class="">
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c<br class="">
b/drivers/gpu/drm/amd/amdkfd/kfd_device.c<br class="">
index 1c2cf3a33c1f..7754f77248a4 100644<br class="">
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c<br class="">
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c<br class="">
@@ -952,11 +952,12 @@ bool kfd_is_locked(void)<br class="">
<br class="">
void kgd2kfd_suspend(struct kfd_dev *kfd, bool run_pm)<br class="">
{<br class="">
+<br class="">
if (!kfd->init_complete)<br class="">
return;<br class="">
<br class="">
/* for runtime suspend, skip locking kfd */<br class="">
- if (!run_pm) {<br class="">
+ if (!run_pm && !drm_dev_is_unplugged(kfd->ddev)) {<br class="">
/* For first KFD device suspend all the KFD processes */<br class="">
if (atomic_inc_return(&kfd_locked) == 1)<br class="">
kfd_suspend_all_processes();<br class="">
<br class="">
<br class="">
Andrey<br class="">
<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">+<br class="">
int kgd2kfd_resume_iommu(struct kfd_dev *kfd)<br class="">
{<br class="">
int err = 0;</blockquote>
</blockquote>
</blockquote>
</div>
</blockquote>
</div>
<br class="">
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>