<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""> <br class=""> <div><br class=""> <blockquote type="cite" class=""> <div class="">On Apr 11, 2022, at 11:52 PM, Andrey Grodzovsky <<a href="mailto:andrey.grodzovsky@amd.com" class="">andrey.grodzovsky@amd.com</a>> wrote:</div> <br class="Apple-interchange-newline"> <div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">[Some people who received this message don't often get email from<span class="Apple-converted-space"> </span></span><a href="mailto:andrey.grodzovsky@amd.com" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">andrey.grodzovsky@amd.com</a><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">. Learn why this is important at<span class="Apple-converted-space"> </span></span><a href="http://aka.ms/LearnAboutSenderIdentification" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://aka.ms/LearnAboutSenderIdentification</a><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">.]</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">On 2022-04-08 21:28, Shuotao Xu wrote:</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br class=""> <blockquote type="cite" class="">On Apr 8, 2022, at 11:28 PM, Andrey Grodzovsky <<a href="mailto:andrey.grodzovsky@amd.com" class="">andrey.grodzovsky@amd.com</a>> wrote:<br class=""> <br class=""> [Some people who received this message don't often get email from <a href="mailto:andrey.grodzovsky@amd.com" class=""> andrey.grodzovsky@amd.com</a>. Learn why this is important at <a href="http://aka.ms/LearnAboutSenderIdentification" class=""> http://aka.ms/LearnAboutSenderIdentification</a>.]<br class=""> <br class=""> On 2022-04-08 04:45, Shuotao Xu wrote:<br class=""> <blockquote type="cite" class="">Adding PCIe Hotplug Support for AMDKFD: the support of hot-plug of GPU<br class=""> devices can open doors for many advanced applications in data center<br class=""> in the next few years, such as for GPU resource<br class=""> disaggregation. Current AMDKFD does not support hotplug out b/o the<br class=""> following reasons:<br class=""> <br class=""> 1. During PCIe removal, decrement KFD lock which was incremented at<br class=""> the beginning of hw fini; otherwise kfd_open later is going to<br class=""> fail.<br class=""> </blockquote> I assumed you read my comment last time, still you do same approach.<br class=""> More in details bellow<br class=""> </blockquote> Aha, I like your fix:) I was not familiar with drm APIs so just only half understood your comment last time.<br class=""> <br class=""> BTW, I tried hot-plugging out a GPU when rocm application is still running.<br class=""> From dmesg, application is still trying to access the removed kfd device, and are met with some errors.<br class=""> </blockquote> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Application us supposed to keep running, it holds the drm_device</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">reference as long as it has an open</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">FD to the device and final cleanup will come only after the app will die</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">thus releasing the FD and the last</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">drm_device reference.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> Application would hang and not exiting in this case.<br class=""> </blockquote> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> </div> </blockquote> <div><br class=""> </div> Actually I tried kill -7 $pid, and the process exists. The dmesg has some warning though.</div> <div><br class=""> </div> <div> <div>[ 711.769977] WARNING: CPU: 23 PID: 344 at .../amdgpu-rocm5.0.2/src/amd/amdgpu/amdgpu_object.c:1336 amdgpu_bo_release_notify+0x150/0x160 [amdgpu]</div> <div>[ 711.770528] Modules linked in: amdgpu(OE) amdttm(OE) amd_sched(OE) amdkcl(OE) iommu_v2 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay binfmt_misc intel_rapl_msr i40iw intel_rapl_common skx_edac nfit x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel rpcrdma kvm sunrpc ipmi_ssif ib_umad ib_ipoib rdma_ucm irqbypass rapl joydev acpi_ipmi input_leds intel_cstate ipmi_si ipmi_devintf mei_me mei intel_pch_thermal ipmi_msghandler ioatdma mac_hid lpc_ich dca acpi_power_meter acpi_pad sch_fq_codel ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi pci_stub ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor</div> <div>[ 711.779359] raid6_pq libcrc32c raid1 raid0 multipath linear mlx5_ib ib_uverbs ib_core ast drm_vram_helper i2c_algo_bit drm_ttm_helper ttm drm_kms_helper syscopyarea crct10dif_pclmul crc32_pclmul ghash_clmulni_intel sysfillrect uas hid_generic sysimgblt aesni_intel mlx5_core fb_sys_fops crypto_simd usbhid cryptd drm i40e pci_hyperv_intf usb_storage glue_helper mlxfw hid ahci libahci wmi</div> <div>[ 711.779752] CPU: 23 PID: 344 Comm: kworker/23:1 Tainted: G W OE 5.11.0+ #1</div> <div>[ 711.779755] Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 2.1 08/14/2018</div> <div>[ 711.779756] Workqueue: kfd_process_wq kfd_process_wq_release [amdgpu]</div> <div>[ 711.779955] RIP: 0010:amdgpu_bo_release_notify+0x150/0x160 [amdgpu]</div> <div>[ 711.780141] Code: e8 b5 af 34 f4 e9 1f ff ff ff 48 39 c2 74 07 0f 0b e9 69 ff ff ff 4c 89 e7 e8 3c b4 16 00 e9 5c ff ff ff e8 a2 ce fd f3 eb cf <0f> 0b eb cb e8 d7 02 34 f4 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55</div> <div>[ 711.780143] RSP: 0018:ffffa8100dd67c30 EFLAGS: 00010282</div> <div>[ 711.780145] RAX: 00000000ffffffea RBX: ffff89980e792058 RCX: 0000000000000000</div> <div>[ 711.780147] RDX: 0000000000000000 RSI: ffff89a8f9ad8870 RDI: ffff89a8f9ad8870</div> <div>[ 711.780148] RBP: ffffa8100dd67c50 R08: 0000000000000000 R09: fffffffffff99b18</div> <div>[ 711.780149] R10: ffffa8100dd67bd0 R11: ffffa8100dd67908 R12: ffff89980e792000</div> <div>[ 711.780151] R13: ffff89980e792058 R14: ffff89980e7921bc R15: dead000000000100</div> <div>[ 711.780152] FS: 0000000000000000(0000) GS:ffff89a8f9ac0000(0000) knlGS:0000000000000000</div> <div>[ 711.780154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033</div> <div>[ 711.780156] CR2: 00007ffddac6f71f CR3: 00000030bb80a003 CR4: 00000000007706e0</div> <div>[ 711.780157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000</div> <div>[ 711.780159] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400</div> <div>[ 711.780160] PKRU: 55555554</div> <div>[ 711.780161] Call Trace:</div> <div>[ 711.780163] ttm_bo_release+0x2ae/0x320 [amdttm]</div> <div>[ 711.780169] amdttm_bo_put+0x30/0x40 [amdttm]</div> <div>[ 711.780357] amdgpu_bo_unref+0x1e/0x30 [amdgpu]</div> <div>[ 711.780543] amdgpu_gem_object_free+0x8b/0x160 [amdgpu]</div> <div>[ 711.781119] drm_gem_object_free+0x1d/0x30 [drm]</div> <div>[ 711.781489] amdgpu_amdkfd_gpuvm_free_memory_of_gpu+0x34a/0x380 [amdgpu]</div> <div>[ 711.782044] kfd_process_device_free_bos+0xe0/0x130 [amdgpu]</div> <div>[ 711.782611] kfd_process_wq_release+0x286/0x380 [amdgpu]</div> <div>[ 711.783172] process_one_work+0x236/0x420</div> <div>[ 711.783543] worker_thread+0x34/0x400</div> <div>[ 711.783911] ? process_one_work+0x420/0x420</div> <div>[ 711.784279] kthread+0x126/0x140</div> <div>[ 711.784653] ? kthread_park+0x90/0x90</div> <div>[ 711.785018] ret_from_fork+0x22/0x30</div> <div>[ 711.785387] ---[ end trace d8f50f6594817c84 ]---</div> <div>[ 711.798716] [drm] amdgpu: ttm finalized</div> <div><br class=""> </div> </div> <div> <blockquote type="cite" class=""> <div class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">For graphic apps what i usually see is a crash because of sigsev when</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">the app tries to access</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">an unmapped MMIO region on the device. I haven't tested for compute</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">stack and so there might</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">be something I haven't covered. Hang could mean for example waiting on a</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">fence which is not being</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">signaled - please provide full dmesg from this case.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br class=""> Do you have any good suggestions on how to fix it down the line? (HIP runtime/libhsakmt or driver)<br class=""> <br class=""> [64036.631333] amdgpu: amdgpu_vm_bo_update failed<br class=""> [64036.631702] amdgpu: validate_invalid_user_pages: update PTE failed<br class=""> [64036.640754] amdgpu: amdgpu_vm_bo_update failed<br class=""> [64036.641120] amdgpu: validate_invalid_user_pages: update PTE failed<br class=""> [64036.650394] amdgpu: amdgpu_vm_bo_update failed<br class=""> [64036.650765] amdgpu: validate_invalid_user_pages: update PTE failed<br class=""> </blockquote> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> </div> </blockquote> <div><br class=""> </div> The full dmesg will just the repetition of those two messages,</div> <div> <div>[186885.764079] amdgpu 0000:43:00.0: amdgpu: amdgpu: finishing device.</div> <div>[186885.766916] [drm] free PSP TMR buffer</div> <div>[186893.868173] amdgpu: amdgpu_vm_bo_update failed</div> <div>[186893.868235] amdgpu: validate_invalid_user_pages: update PTE failed</div> <div>[186893.876154] amdgpu: amdgpu_vm_bo_update failed</div> <div>[186893.876190] amdgpu: validate_invalid_user_pages: update PTE failed</div> <div>[186893.884150] amdgpu: amdgpu_vm_bo_update failed</div> <div>[186893.884185] amdgpu: validate_invalid_user_pages: update PTE failed</div> </div> <div><br class=""> </div> <div> <blockquote type="cite" class=""> <div class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">This just probably means trying to update PTEs after the physical device</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">is gone - we usually avoid this by</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">first trying to do all HW shutdowns early before PCI remove completion</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">but when it's really tricky by</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">protecting HW access sections with drm_dev_enter/exit scope.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">For this particular error it would be the best to flush</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">info->restore_userptr_work before the end of</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">amdgpu_pci_remove (rejecting new process creation and calling</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">cancel_delayed_work_sync(&process_info->restore_userptr_work) for all</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">running processes)</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">somewhere in amdgpu_pci_remove.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br class=""> </div> </blockquote> I tried something like *kfd_process_ref_release* which I think did what you described, but it did not work.</div> <div><br class=""> </div> <div>Instead I tried to kill the process from the kernel, but the amdgpu could **only** be hot-plugged in back successfully only if there was no rocm kernel running when it was plugged out. If not, amdgpu_probe will just hang later. (Maybe because amdgpu was plugged out while running state, it leaves a bad HW state which causes probe to hang).</div> <div><br class=""> </div> <div>I don’t know if this is a viable solution worth pursuing, but I attached the diff anyway.</div> <div><br class=""> </div> <div>Another solution could be let compute stack user mode detect a topology change via <span class="">generation_count change, and abort gracefully there.</span></div> <div class=""><br class=""> </div> <div>diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c</div> <div>index 4e7d9cb09a69..79b4c9b84cd0 100644</div> <div>--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c</div> <div>+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c</div> <div>@@ -697,12 +697,15 @@ void kgd2kfd_suspend(struct kfd_dev *kfd, bool run_pm, bool force)</div> <div> return;</div> <div><br class=""> </div> <div> /* for runtime suspend, skip locking kfd */</div> <div>- if (!run_pm) {</div> <div>+ if (!run_pm && !drm_dev_is_unplugged(kfd->ddev)) {</div> <div> /* For first KFD device suspend all the KFD processes */</div> <div> if (atomic_inc_return(&kfd_locked) == 1)</div> <div> kfd_suspend_all_processes(force);</div> <div> }</div> <div><br class=""> </div> <div>+ if (drm_dev_is_unplugged(kfd->ddev))</div> <div>+ kfd_kill_all_user_processes();</div> <div>+</div> <div> kfd->dqm->ops.stop(kfd->dqm);</div> <div> kfd_iommu_suspend(kfd);</div> <div> }</div> <div>diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h</div> <div>index 55c9e1922714..84cbcd857856 100644</div> <div>--- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h</div> <div>+++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h</div> <div>@@ -1065,6 +1065,7 @@ void kfd_unref_process(struct kfd_process *p);</div> <div> int kfd_process_evict_queues(struct kfd_process *p, bool force);</div> <div> int kfd_process_restore_queues(struct kfd_process *p);</div> <div> void kfd_suspend_all_processes(bool force);</div> <div>+void kfd_kill_all_user_processes(void);</div> <div> /*</div> <div> * kfd_resume_all_processes:</div> <div> * bool sync: If kfd_resume_all_processes() should wait for the</div> <div>diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c</div> <div>index 6cdc855abb6d..fb0c753b682c 100644</div> <div>--- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c</div> <div>+++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c</div> <div>@@ -2206,6 +2206,24 @@ void kfd_suspend_all_processes(bool force)</div> <div> srcu_read_unlock(&kfd_processes_srcu, idx);</div> <div> }</div> <div><br class=""> </div> <div>+</div> <div>+void kfd_kill_all_user_processes(void)</div> <div>+{</div> <div>+ struct kfd_process *p;</div> <div>+ struct amdkfd_process_info *p_info;</div> <div>+ unsigned int temp;</div> <div>+ int idx = srcu_read_lock(&kfd_processes_srcu);</div> <div>+</div> <div>+ pr_info("Killing all processes\n");</div> <div>+ hash_for_each_rcu(kfd_processes_table, temp, p, kfd_processes) {</div> <div>+ p_info = p->kgd_process_info;</div> <div>+ pr_info("Killing processes, pid = %d", pid_nr(p_info->pid));</div> <div>+ kill_pid(p_info->pid, SIGBUS, 1);</div> <div>+ }</div> <div>+ srcu_read_unlock(&kfd_processes_srcu, idx);</div> <div>+}</div> <div>+</div> <div>+</div> <div> int kfd_resume_all_processes(bool sync)</div> <div> {</div> <div> struct kfd_process *p;</div> <div><br class=""> </div> <div><br class=""> <blockquote type="cite" class=""> <div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Andrey</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br class=""> Really appreciate your help!<br class=""> <br class=""> Best,<br class=""> Shuotao<br class=""> <br class=""> <blockquote type="cite" class=""> <blockquote type="cite" class="">2. Remove redudant p2p/io links in sysfs when device is hotplugged<br class=""> out.<br class=""> <br class=""> 3. New kfd node_id is not properly assigned after a new device is<br class=""> added after a gpu is hotplugged out in a system. libhsakmt will<br class=""> find this anomaly, (i.e. node_from != <dev node id> in iolinks),<br class=""> when taking a topology_snapshot, thus returns fault to the rocm<br class=""> stack.<br class=""> <br class=""> -- This patch fixes issue 1; another patch by Mukul fixes issues 2&3.<br class=""> -- Tested on a 4-GPU MI100 gpu nodes with kernel 5.13.0-kfd; kernel<br class=""> 5.16.0-kfd is unstable out of box for MI100.<br class=""> ---<br class=""> drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 5 +++++<br class=""> drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 7 +++++++<br class=""> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 +<br class=""> drivers/gpu/drm/amd/amdkfd/kfd_device.c | 13 +++++++++++++<br class=""> 4 files changed, 26 insertions(+)<br class=""> <br class=""> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c<br class=""> index c18c4be1e4ac..d50011bdb5c4 100644<br class=""> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c<br class=""> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c<br class=""> @@ -213,6 +213,11 @@ int amdgpu_amdkfd_resume(struct amdgpu_device *adev, bool run_pm)<br class=""> return r;<br class=""> }<br class=""> <br class=""> +int amdgpu_amdkfd_resume_processes(void)<br class=""> +{<br class=""> + return kgd2kfd_resume_processes();<br class=""> +}<br class=""> +<br class=""> int amdgpu_amdkfd_pre_reset(struct amdgpu_device *adev)<br class=""> {<br class=""> int r = 0;<br class=""> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h<br class=""> index f8b9f27adcf5..803306e011c3 100644<br class=""> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h<br class=""> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h<br class=""> @@ -140,6 +140,7 @@ void amdgpu_amdkfd_fini(void);<br class=""> void amdgpu_amdkfd_suspend(struct amdgpu_device *adev, bool run_pm);<br class=""> int amdgpu_amdkfd_resume_iommu(struct amdgpu_device *adev);<br class=""> int amdgpu_amdkfd_resume(struct amdgpu_device *adev, bool run_pm);<br class=""> +int amdgpu_amdkfd_resume_processes(void);<br class=""> void amdgpu_amdkfd_interrupt(struct amdgpu_device *adev,<br class=""> const void *ih_ring_entry);<br class=""> void amdgpu_amdkfd_device_probe(struct amdgpu_device *adev);<br class=""> @@ -347,6 +348,7 @@ void kgd2kfd_device_exit(struct kfd_dev *kfd);<br class=""> void kgd2kfd_suspend(struct kfd_dev *kfd, bool run_pm);<br class=""> int kgd2kfd_resume_iommu(struct kfd_dev *kfd);<br class=""> int kgd2kfd_resume(struct kfd_dev *kfd, bool run_pm);<br class=""> +int kgd2kfd_resume_processes(void);<br class=""> int kgd2kfd_pre_reset(struct kfd_dev *kfd);<br class=""> int kgd2kfd_post_reset(struct kfd_dev *kfd);<br class=""> void kgd2kfd_interrupt(struct kfd_dev *kfd, const void *ih_ring_entry);<br class=""> @@ -393,6 +395,11 @@ static inline int kgd2kfd_resume(struct kfd_dev *kfd, bool run_pm)<br class=""> return 0;<br class=""> }<br class=""> <br class=""> +static inline int kgd2kfd_resume_processes(void)<br class=""> +{<br class=""> + return 0;<br class=""> +}<br class=""> +<br class=""> static inline int kgd2kfd_pre_reset(struct kfd_dev *kfd)<br class=""> {<br class=""> return 0;<br class=""> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br class=""> index fa4a9f13c922..5827b65b7489 100644<br class=""> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br class=""> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c<br class=""> @@ -4004,6 +4004,7 @@ void amdgpu_device_fini_hw(struct amdgpu_device *adev)<br class=""> if (drm_dev_is_unplugged(adev_to_drm(adev)))<br class=""> amdgpu_device_unmap_mmio(adev);<br class=""> <br class=""> + amdgpu_amdkfd_resume_processes();<br class=""> }<br class=""> <br class=""> void amdgpu_device_fini_sw(struct amdgpu_device *adev)<br class=""> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c<br class=""> index 62aa6c9d5123..ef05aae9255e 100644<br class=""> --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c<br class=""> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c<br class=""> @@ -714,6 +714,19 @@ int kgd2kfd_resume(struct kfd_dev *kfd, bool run_pm)<br class=""> return ret;<br class=""> }<br class=""> <br class=""> +/* for non-runtime resume only */<br class=""> +int kgd2kfd_resume_processes(void)<br class=""> +{<br class=""> + int count;<br class=""> +<br class=""> + count = atomic_dec_return(&kfd_locked);<br class=""> + WARN_ONCE(count < 0, "KFD suspend / resume ref. error");<br class=""> + if (count == 0)<br class=""> + return kfd_resume_all_processes();<br class=""> +<br class=""> + return 0;<br class=""> +}<br class=""> </blockquote> <br class=""> It doesn't make sense to me to just increment kfd_locked in<br class=""> kgd2kfd_suspend to only decrement it again a few functions down the<br class=""> road.<br class=""> <br class=""> I suggest this instead - you only incrmemnt if not during PCI remove<br class=""> <br class=""> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c<br class=""> b/drivers/gpu/drm/amd/amdkfd/kfd_device.c<br class=""> index 1c2cf3a33c1f..7754f77248a4 100644<br class=""> --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c<br class=""> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c<br class=""> @@ -952,11 +952,12 @@ bool kfd_is_locked(void)<br class=""> <br class=""> void kgd2kfd_suspend(struct kfd_dev *kfd, bool run_pm)<br class=""> {<br class=""> +<br class=""> if (!kfd->init_complete)<br class=""> return;<br class=""> <br class=""> /* for runtime suspend, skip locking kfd */<br class=""> - if (!run_pm) {<br class=""> + if (!run_pm && !drm_dev_is_unplugged(kfd->ddev)) {<br class=""> /* For first KFD device suspend all the KFD processes */<br class=""> if (atomic_inc_return(&kfd_locked) == 1)<br class=""> kfd_suspend_all_processes();<br class=""> <br class=""> <br class=""> Andrey<br class=""> <br class=""> <br class=""> <br class=""> <blockquote type="cite" class="">+<br class=""> int kgd2kfd_resume_iommu(struct kfd_dev *kfd)<br class=""> {<br class=""> int err = 0;</blockquote> </blockquote> </blockquote> </div> </blockquote> </div> <br class=""> </body> </html>