<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=Windows-1252"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> Hi Joshi,<o:p></o:p> <o:p> </o:p> Per your comment, I produced a fix to work with multi-GPU system for hotplug support for our group’s internal usage.<o:p></o:p> <o:p> </o:p> I have tested on a 4-node MI100 system, which seems to be working. It is pushed in the github PR. <o:p></o:p> The details are in: <a href="https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/pull/131#issuecomment-1091843803"> https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/pull/131#issuecomment-1091843803</a><o:p></o:p> <o:p> </o:p> I will send the patch to the mail-list also.<o:p></o:p> <o:p> </o:p> May I know when your patch is ready for public review?<o:p></o:p> <o:p> </o:p> All the best,<o:p></o:p> Shuotao<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> From: Joshi, Mukul <Mukul.Joshi@amd.com> Date: Thursday, April 7, 2022 at 10:24 AM To: Shuotao Xu <shuotaoxu@microsoft.com>, Grodzovsky, Andrey <Andrey.Grodzovsky@amd.com>, amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org> Cc: Ziyue Yang <Ziyue.Yang@microsoft.com>, Lei Qu <Lei.Qu@microsoft.com>, Peng Cheng <pengc@microsoft.com>, Ran Shu <Ran.Shu@microsoft.com> Subject: RE: [EXTERNAL] Re: Code Review Request for AMDGPU Hotplug Support<o:p></o:p> </div> <table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%;cellpadding:0;cellspacing:0;display:table;border-collapse:seperate;float:none"> <tbody> <tr> <td style="background:#A6A6A6;padding:5.25pt 1.5pt 5.25pt 1.5pt;valign:middle"></td> <td width="100%" style="width:100.0%;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 11.25pt"> <div> You don't often get email from mukul.joshi@amd.com. <a href="http://aka.ms/LearnAboutSenderIdentification">Learn why this is important</a><o:p></o:p> </div> </td> <td width="75" style="width:56.25pt;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 3.75pt;align:left"> </td> </tr> </tbody> </table> <div> [AMD Official Use Only]<o:p></o:p> <o:p></o:p> Hi Shuotao,<o:p></o:p> <o:p></o:p> Thanks for your patch.<o:p></o:p> I have been working on something similar as I also found that we don’t cleanup IO links upon device removal.<o:p></o:p> <o:p></o:p> The IO-links cleanup change in your patch would work only either on a single GPU system or a multi-GPU system where the last node (in the sysfs topology) is hot-plugged out. That’s because of the way the atomic counter, topology_crat_proximity_domain, is used in the code.<o:p></o:p> <o:p></o:p> I have a patch which takes care of these issues on a multi-GPU system.<o:p></o:p> I should be able to send that out for review in sometime.<o:p></o:p> <o:p></o:p> Thanks,<o:p></o:p> Mukul<o:p></o:p> <o:p></o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Shuotao Xu Sent: Wednesday, April 6, 2022 11:12 AM To: Grodzovsky, Andrey <Andrey.Grodzovsky@amd.com>; amd-gfx@lists.freedesktop.org Cc: Ziyue Yang <Ziyue.Yang@microsoft.com>; Lei Qu <Lei.Qu@microsoft.com>; Peng Cheng <pengc@microsoft.com>; Ran Shu <Ran.Shu@microsoft.com> Subject: Re: [EXTERNAL] Re: Code Review Request for AMDGPU Hotplug Support<o:p></o:p> </div> </div> <o:p></o:p> [CAUTION: External Email] <o:p></o:p> <div> Hi Andrey,<o:p></o:p> <o:p></o:p> Thanks for your kind comment on linux patch submission protocol, please let me know if there is anyway to rectify it. <o:p></o:p> <o:p></o:p> dmesg is fine except with some warning during pci rescan after pci removal of an AMD MI100. <o:p></o:p> <o:p></o:p> The issue is that after this rocm application will return segfault with the amdgpu driver unless the entire amdgpu kernel module is unloaded and loaded, which we did not meet our hotplug requirement. The issues upon investigation are<o:p></o:p> 1) kfd_lock is locked after hotplug, and kfd_open will return fault right away to libhsakmt . <o:p></o:p> 2) iolink/p2plink has anormalies after hotplug, and libhsakmt will found such anomalies and return error.<o:p></o:p> <o:p></o:p> Our patch has been tested with a single-instance AMD MI100 GPU and showed it worked.<o:p></o:p> <o:p></o:p> I am attaching the dmesg after rescan anyway, which will show the warning and the segfault.<o:p></o:p> <o:p></o:p> [ 132.054822] pci 0000:43:00.0: [1002:738c] type 00 class 0x038000<o:p></o:p> [ 132.054856] pci 0000:43:00.0: reg 0x10: [mem 0x38b000000000-0x38b7ffffffff 64bit pref]<o:p></o:p> [ 132.054877] pci 0000:43:00.0: reg 0x18: [mem 0x38b800000000-0x38b8001fffff 64bit pref]<o:p></o:p> [ 132.054890] pci 0000:43:00.0: reg 0x20: [io 0xa000-0xa0ff]<o:p></o:p> [ 132.054904] pci 0000:43:00.0: reg 0x24: [mem 0xb8400000-0xb847ffff]<o:p></o:p> [ 132.054918] pci 0000:43:00.0: reg 0x30: [mem 0xb8480000-0xb849ffff pref]<o:p></o:p> [ 132.055134] pci 0000:43:00.0: PME# supported from D1 D2 D3hot D3cold<o:p></o:p> [ 132.055217] pci 0000:43:00.0: 63.008 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x8 link at 0000:3c:14.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)<o:p></o:p> [ 132.056001] pci 0000:43:00.0: Adding to iommu group 73<o:p></o:p> [ 132.057943] pci 0000:43:00.0: BAR 0: assigned [mem 0x38b000000000-0x38b7ffffffff 64bit pref]<o:p></o:p> [ 132.057960] pci 0000:43:00.0: BAR 2: assigned [mem 0x38b800000000-0x38b8001fffff 64bit pref]<o:p></o:p> [ 132.057974] pci 0000:43:00.0: BAR 5: assigned [mem 0xb8400000-0xb847ffff]<o:p></o:p> [ 132.057981] pci 0000:43:00.0: BAR 6: assigned [mem 0xb8480000-0xb849ffff pref]<o:p></o:p> [ 132.057984] pci 0000:43:00.0: BAR 4: assigned [io 0xa000-0xa0ff]<o:p></o:p> <o:p></o:p> [ 132.058429] ======================================================<o:p></o:p> [ 132.058453] WARNING: possible circular locking dependency detected<o:p></o:p> [ 132.058477] 5.16.0-kfd+ #1 Not tainted<o:p></o:p> [ 132.058492] ------------------------------------------------------<o:p></o:p> [ 132.058515] bash/3632 is trying to acquire lock:<o:p></o:p> [ 132.058534] ffffadee20adfb50 ((work_completion)(&wfc.work)){+.+.}-{0:0}, at: __flush_work+0x2f5/0x470<o:p></o:p> [ 132.058554] [drm] initializing kernel modesetting (ARCTURUS 0x1002:0x738C 0x1002:0x0C34 0x01).<o:p></o:p> [ 132.058577]<o:p></o:p> but task is already holding lock:<o:p></o:p> [ 132.058580] ffffffffa3c62308<o:p></o:p> [ 132.058630] amdgpu 0000:43:00.0: amdgpu: Trusted Memory Zone (TMZ) feature not supported<o:p></o:p> [ 132.058638] (<o:p></o:p> [ 132.058678] [drm] register mmio base: 0xB8400000<o:p></o:p> [ 132.058683] pci_rescan_remove_lock<o:p></o:p> [ 132.058694] [drm] register mmio size: 524288<o:p></o:p> [ 132.058713] ){+.+.}-{3:3}, at: rescan_store+0x55/0x90<o:p></o:p> [ 132.058773]<o:p></o:p> which lock already depends on the new lock.<o:p></o:p> <o:p></o:p> [ 132.058804]<o:p></o:p> the existing dependency chain (in reverse order) is:<o:p></o:p> [ 132.058819] [drm] add ip block number 0 <soc15_common><o:p></o:p> [ 132.058831]<o:p></o:p> -> #1 (<o:p></o:p> [ 132.058854] [drm] add ip block number 1 <gmc_v9_0><o:p></o:p> [ 132.058858] [drm] add ip block number 2 <vega20_ih><o:p></o:p> [ 132.058874] pci_rescan_remove_lock<o:p></o:p> [ 132.058894] [drm] add ip block number 3 <psp><o:p></o:p> [ 132.058915] ){+.+.}-{3:3}<o:p></o:p> [ 132.058931] [drm] add ip block number 4 <smu><o:p></o:p> [ 132.058951] :<o:p></o:p> [ 132.058965] [drm] add ip block number 5 <gfx_v9_0><o:p></o:p> [ 132.058986] __mutex_lock+0xa4/0x990<o:p></o:p> [ 132.058996] [drm] add ip block number 6 <sdma_v4_0><o:p></o:p> [ 132.059016] i801_add_tco_spt.isra.20+0x2a/0x1a0<o:p></o:p> [ 132.059033] [drm] add ip block number 7 <vcn_v2_5><o:p></o:p> [ 132.059054] i801_add_tco+0xf6/0x110<o:p></o:p> [ 132.059075] [drm] add ip block number 8 <jpeg_v2_5><o:p></o:p> [ 132.059096] i801_probe+0x402/0x860<o:p></o:p> [ 132.059151] local_pci_probe+0x40/0x90<o:p></o:p> [ 132.059170] work_for_cpu_fn+0x10/0x20<o:p></o:p> [ 132.059189] process_one_work+0x2a4/0x640<o:p></o:p> [ 132.059208] worker_thread+0x228/0x3f0<o:p></o:p> [ 132.059227] kthread+0x16d/0x1a0<o:p></o:p> [ 132.059795] ret_from_fork+0x1f/0x30<o:p></o:p> [ 132.060337]<o:p></o:p> -> #0 ((work_completion)(&wfc.work)){+.+.}-{0:0}:<o:p></o:p> [ 132.061405] __lock_acquire+0x1552/0x1ac0<o:p></o:p> [ 132.061950] lock_acquire+0x26c/0x300<o:p></o:p> [ 132.062484] __flush_work+0x315/0x470<o:p></o:p> [ 132.063009] work_on_cpu+0x98/0xc0<o:p></o:p> [ 132.063526] pci_device_probe+0x1bc/0x1d0<o:p></o:p> [ 132.064036] really_probe+0x102/0x450<o:p></o:p> [ 132.064532] __driver_probe_device+0x100/0x170<o:p></o:p> [ 132.065020] driver_probe_device+0x1f/0xa0<o:p></o:p> [ 132.065497] __device_attach_driver+0x6b/0xe0<o:p></o:p> [ 132.065975] bus_for_each_drv+0x6a/0xb0<o:p></o:p> [ 132.066449] __device_attach+0xe2/0x160<o:p></o:p> [ 132.066912] pci_bus_add_device+0x4a/0x80<o:p></o:p> [ 132.067365] pci_bus_add_devices+0x2c/0x70<o:p></o:p> [ 132.067812] pci_bus_add_devices+0x65/0x70<o:p></o:p> [ 132.068253] pci_bus_add_devices+0x65/0x70<o:p></o:p> [ 132.068688] pci_bus_add_devices+0x65/0x70<o:p></o:p> [ 132.068936] amdgpu 0000:43:00.0: amdgpu: Fetched VBIOS from ROM BAR<o:p></o:p> [ 132.069109] pci_bus_add_devices+0x65/0x70<o:p></o:p> [ 132.069602] amdgpu: ATOM BIOS: 113-D3431401-X00<o:p></o:p> [ 132.070058] pci_bus_add_devices+0x65/0x70<o:p></o:p> [ 132.070572] [drm] VCN(0) decode is enabled in VM mode<o:p></o:p> [ 132.070997] pci_rescan_bus+0x23/0x30<o:p></o:p> [ 132.071000] rescan_store+0x61/0x90<o:p></o:p> [ 132.071003] kernfs_fop_write_iter+0x132/0x1b0<o:p></o:p> [ 132.071501] [drm] VCN(1) decode is enabled in VM mode<o:p></o:p> [ 132.071964] new_sync_write+0x11f/0x1b0<o:p></o:p> [ 132.072432] [drm] VCN(0) encode is enabled in VM mode<o:p></o:p> [ 132.072900] vfs_write+0x35b/0x3b0<o:p></o:p> [ 132.073376] [drm] VCN(1) encode is enabled in VM mode<o:p></o:p> [ 132.073847] ksys_write+0xa7/0xe0<o:p></o:p> [ 132.074335] [drm] JPEG(0) JPEG decode is enabled in VM mode<o:p></o:p> [ 132.074803] do_syscall_64+0x34/0x80<o:p></o:p> [ 132.074808] entry_SYSCALL_64_after_hwframe+0x44/0xae<o:p></o:p> [ 132.074811]<o:p></o:p> other info that might help us debug this:<o:p></o:p> <o:p></o:p> [ 132.074813] Possible unsafe locking scenario:<o:p></o:p> <o:p></o:p> [ 132.075302] [drm] JPEG(1) JPEG decode is enabled in VM mode<o:p></o:p> [ 132.075779] CPU0 CPU1<o:p></o:p> [ 132.076361] amdgpu 0000:43:00.0: amdgpu: MEM ECC is active.<o:p></o:p> [ 132.076765] ---- ----<o:p></o:p> [ 132.077265] amdgpu 0000:43:00.0: amdgpu: SRAM ECC is active.<o:p></o:p> [ 132.078649] lock(pci_rescan_remove_lock);<o:p></o:p> [ 132.078652] lock((work_completion)(&wfc.work));<o:p></o:p> [ 132.078653] lock(pci_rescan_remove_lock);<o:p></o:p> [ 132.078655] lock((work_completion)(&wfc.work));<o:p></o:p> [ 132.078656]<o:p></o:p> *** DEADLOCK ***<o:p></o:p> <o:p></o:p> [ 132.078656] 5 locks held by bash/3632:<o:p></o:p> [ 132.078658] #0: ffff9c39c7b89438<o:p></o:p> [ 132.079612] amdgpu 0000:43:00.0: amdgpu: RAS INFO: ras initialized successfully, hardware ability[7fff] ras_mask[7fff]<o:p></o:p> [ 132.080089] (<o:p></o:p> [ 132.080602] [drm] vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit<o:p></o:p> [ 132.081082] sb_writers<o:p></o:p> [ 132.081601] amdgpu 0000:43:00.0: amdgpu: VRAM: 32752M 0x0000008000000000 - 0x00000087FEFFFFFF (32752M used)<o:p></o:p> [ 132.082102] #6<o:p></o:p> [ 132.082630] amdgpu 0000:43:00.0: amdgpu: GART: 512M 0x0000000000000000 - 0x000000001FFFFFFF<o:p></o:p> [ 132.083152] ){.+.+}-{0:0}<o:p></o:p> [ 132.083687] amdgpu 0000:43:00.0: amdgpu: AGP: 267878400M 0x0000008800000000 - 0x0000FFFFFFFFFFFF<o:p></o:p> [ 132.084210] , at: ksys_write+0xa7/0xe0<o:p></o:p> [ 132.085708] [drm] Detected VRAM RAM=32752M, BAR=32768M<o:p></o:p> [ 132.086205] #1:<o:p></o:p> [ 132.086733] [drm] RAM width 4096bits HBM<o:p></o:p> [ 132.087269] ffff9c5959011088<o:p></o:p> [ 132.087890] [drm] amdgpu: 32752M of VRAM memory ready<o:p></o:p> [ 132.088389] (<o:p></o:p> [ 132.088972] [drm] amdgpu: 32752M of GTT memory ready.<o:p></o:p> [ 132.089572] &of->mutex<o:p></o:p> [ 132.090206] [drm] GART: num cpu pages 131072, num gpu pages 131072<o:p></o:p> [ 132.090804] ){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x103/0x1b0<o:p></o:p> [ 132.090808] #2: ffff9c39c882a9e0 (kn->active#423){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x10c/0x1b0<o:p></o:p> [ 132.091639] [drm] PCIE GART of 512M enabled.<o:p></o:p> [ 132.092117] #3:<o:p></o:p> [ 132.092801] [drm] PTB located at 0x0000008000000000<o:p></o:p> [ 132.093480] ffffffffa3c62308<o:p></o:p> [ 132.094566] amdgpu 0000:43:00.0: amdgpu: PSP runtime database doesn't exist<o:p></o:p> [ 132.094822] (pci_rescan_remove_lock){+.+.}-{3:3}, at: rescan_store+0x55/0x90<o:p></o:p> [ 132.094827] #4: ffff9c597392b248 (&dev->mutex){....}-{3:3}, at: __device_attach+0x39/0x160<o:p></o:p> [ 132.094835]<o:p></o:p> stack backtrace:<o:p></o:p> [ 132.097098] [drm] Found VCN firmware Version ENC: 1.1 DEC: 1 VEP: 0 Revision: 21<o:p></o:p> [ 132.097467] CPU: 47 PID: 3632 Comm: bash Not tainted 5.16.0-kfd+ #1<o:p></o:p> [ 132.098169] amdgpu 0000:43:00.0: amdgpu: Will use PSP to load VCN firmware<o:p></o:p> [ 132.098839] Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 2.1 08/14/2018<o:p></o:p> [ 132.098841] Call Trace:<o:p></o:p> [ 132.098842] <TASK><o:p></o:p> [ 132.098843] dump_stack_lvl+0x44/0x57<o:p></o:p> [ 132.098848] check_noncircular+0x105/0x120<o:p></o:p> [ 132.098853] ? unwind_get_return_address+0x1b/0x30<o:p></o:p> [ 132.112924] ? register_lock_class+0x46/0x780<o:p></o:p> [ 132.113630] ? __lock_acquire+0x1552/0x1ac0<o:p></o:p> [ 132.114342] __lock_acquire+0x1552/0x1ac0<o:p></o:p> [ 132.115050] lock_acquire+0x26c/0x300<o:p></o:p> [ 132.115755] ? __flush_work+0x2f5/0x470<o:p></o:p> [ 132.116460] ? lock_is_held_type+0xdf/0x130<o:p></o:p> [ 132.117177] __flush_work+0x315/0x470<o:p></o:p> [ 132.117890] ? __flush_work+0x2f5/0x470<o:p></o:p> [ 132.118604] ? lock_is_held_type+0xdf/0x130<o:p></o:p> [ 132.119305] ? mark_held_locks+0x49/0x70<o:p></o:p> [ 132.119981] ? queue_work_on+0x2f/0x70<o:p></o:p> [ 132.120645] ? lockdep_hardirqs_on+0x79/0x100<o:p></o:p> [ 132.121300] work_on_cpu+0x98/0xc0<o:p></o:p> [ 132.121702] [drm] reserve 0x400000 from 0x87fec00000 for PSP TMR<o:p></o:p> [ 132.121947] ? __traceiter_workqueue_execute_end+0x40/0x40<o:p></o:p> [ 132.123270] ? pci_device_shutdown+0x60/0x60<o:p></o:p> [ 132.123880] pci_device_probe+0x1bc/0x1d0<o:p></o:p> [ 132.124475] really_probe+0x102/0x450<o:p></o:p> [ 132.125060] __driver_probe_device+0x100/0x170<o:p></o:p> [ 132.125641] driver_probe_device+0x1f/0xa0<o:p></o:p> [ 132.126215] __device_attach_driver+0x6b/0xe0<o:p></o:p> [ 132.126797] ? driver_allows_async_probing+0x50/0x50<o:p></o:p> [ 132.127383] ? driver_allows_async_probing+0x50/0x50<o:p></o:p> [ 132.127960] bus_for_each_drv+0x6a/0xb0<o:p></o:p> [ 132.128528] __device_attach+0xe2/0x160<o:p></o:p> [ 132.129095] pci_bus_add_device+0x4a/0x80<o:p></o:p> [ 132.129659] pci_bus_add_devices+0x2c/0x70<o:p></o:p> [ 132.130213] pci_bus_add_devices+0x65/0x70<o:p></o:p> [ 132.130753] pci_bus_add_devices+0x65/0x70<o:p></o:p> [ 132.131283] pci_bus_add_devices+0x65/0x70<o:p></o:p> [ 132.131780] pci_bus_add_devices+0x65/0x70<o:p></o:p> [ 132.132270] pci_bus_add_devices+0x65/0x70<o:p></o:p> [ 132.132757] pci_rescan_bus+0x23/0x30<o:p></o:p> [ 132.133233] rescan_store+0x61/0x90<o:p></o:p> [ 132.133701] kernfs_fop_write_iter+0x132/0x1b0<o:p></o:p> [ 132.134167] new_sync_write+0x11f/0x1b0<o:p></o:p> [ 132.134627] vfs_write+0x35b/0x3b0<o:p></o:p> [ 132.135062] ksys_write+0xa7/0xe0<o:p></o:p> [ 132.135503] do_syscall_64+0x34/0x80<o:p></o:p> [ 132.135933] entry_SYSCALL_64_after_hwframe+0x44/0xae<o:p></o:p> [ 132.136358] RIP: 0033:0x7f0058a73224<o:p></o:p> [ 132.136775] Code: 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 8d 05 c1 07 2e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 f3 c3 66 90 41 54 55 49 89 d4 53 48 89 f5<o:p></o:p> [ 132.137663] RSP: 002b:00007ffc4f4c71a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001<o:p></o:p> [ 132.138121] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f0058a73224<o:p></o:p> [ 132.138590] RDX: 0000000000000002 RSI: 000055d466c24450 RDI: 0000000000000001<o:p></o:p> [ 132.139064] RBP: 000055d466c24450 R08: 000000000000000a R09: 0000000000000001<o:p></o:p> [ 132.139532] R10: 000000000000000a R11: 0000000000000246 R12: 00007f0058d4f760<o:p></o:p> [ 132.140003] R13: 0000000000000002 R14: 00007f0058d4b2a0 R15: 00007f0058d4a760<o:p></o:p> [ 132.140485] </TASK><o:p></o:p> [ 132.183669] amdgpu 0000:43:00.0: amdgpu: HDCP: optional hdcp ta ucode is not available<o:p></o:p> [ 132.184214] amdgpu 0000:43:00.0: amdgpu: DTM: optional dtm ta ucode is not available<o:p></o:p> [ 132.184735] amdgpu 0000:43:00.0: amdgpu: RAP: optional rap ta ucode is not available<o:p></o:p> [ 132.185234] amdgpu 0000:43:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available<o:p></o:p> [ 132.185823] amdgpu 0000:43:00.0: amdgpu: use vbios provided pptable<o:p></o:p> [ 132.186327] amdgpu 0000:43:00.0: amdgpu: smc_dpm_info table revision(format.content): 4.6<o:p></o:p> [ 132.188783] amdgpu 0000:43:00.0: amdgpu: SMU is initialized successfully!<o:p></o:p> [ 132.190039] [drm] kiq ring mec 2 pipe 1 q 0<o:p></o:p> [ 132.203608] [drm] VCN decode and encode initialized successfully(under DPG Mode).<o:p></o:p> [ 132.204178] [drm] JPEG decode initialized successfully.<o:p></o:p> [ 132.246079] kfd kfd: amdgpu: Allocated 3969056 bytes on gart<o:p></o:p> [ 132.327589] memmap_init_zone_device initialised 8388608 pages in 64ms<o:p></o:p> [ 132.328139] amdgpu: HMM registered 32752MB device memory<o:p></o:p> [ 132.328784] amdgpu: Virtual CRAT table created for GPU<o:p></o:p> [ 132.329844] amdgpu: Topology: Add dGPU node [0x738c:0x1002]<o:p></o:p> [ 132.330387] kfd kfd: amdgpu: added device 1002:738c<o:p></o:p> [ 132.330965] amdgpu 0000:43:00.0: amdgpu: SE 8, SH per SE 1, CU per SH 16, active_cu_number 72<o:p></o:p> [ 132.331725] amdgpu 0000:43:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 0 on hub 0<o:p></o:p> [ 132.332296] amdgpu 0000:43:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 1 on hub 0<o:p></o:p> [ 132.332856] amdgpu 0000:43:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 4 on hub 0<o:p></o:p> [ 132.333414] amdgpu 0000:43:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 5 on hub 0<o:p></o:p> [ 132.333965] amdgpu 0000:43:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 6 on hub 0<o:p></o:p> [ 132.334507] amdgpu 0000:43:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 7 on hub 0<o:p></o:p> [ 132.335057] amdgpu 0000:43:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 8 on hub 0<o:p></o:p> [ 132.335594] amdgpu 0000:43:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 9 on hub 0<o:p></o:p> [ 132.336137] amdgpu 0000:43:00.0: amdgpu: ring kiq_2.1.0 uses VM inv eng 10 on hub 0<o:p></o:p> [ 132.336679] amdgpu 0000:43:00.0: amdgpu: ring sdma0 uses VM inv eng 0 on hub 1<o:p></o:p> [ 132.337234] amdgpu 0000:43:00.0: amdgpu: ring sdma1 uses VM inv eng 1 on hub 1<o:p></o:p> [ 132.337790] amdgpu 0000:43:00.0: amdgpu: ring sdma2 uses VM inv eng 4 on hub 1<o:p></o:p> [ 132.338343] amdgpu 0000:43:00.0: amdgpu: ring sdma3 uses VM inv eng 5 on hub 1<o:p></o:p> [ 132.338906] amdgpu 0000:43:00.0: amdgpu: ring sdma4 uses VM inv eng 6 on hub 1<o:p></o:p> [ 132.339448] amdgpu 0000:43:00.0: amdgpu: ring sdma5 uses VM inv eng 0 on hub 2<o:p></o:p> [ 132.339987] amdgpu 0000:43:00.0: amdgpu: ring sdma6 uses VM inv eng 1 on hub 2<o:p></o:p> [ 132.340519] amdgpu 0000:43:00.0: amdgpu: ring sdma7 uses VM inv eng 4 on hub 2<o:p></o:p> [ 132.341041] amdgpu 0000:43:00.0: amdgpu: ring vcn_dec_0 uses VM inv eng 5 on hub 2<o:p></o:p> [ 132.341570] amdgpu 0000:43:00.0: amdgpu: ring vcn_enc_0.0 uses VM inv eng 6 on hub 2<o:p></o:p> [ 132.342101] amdgpu 0000:43:00.0: amdgpu: ring vcn_enc_0.1 uses VM inv eng 7 on hub 2<o:p></o:p> [ 132.342630] amdgpu 0000:43:00.0: amdgpu: ring vcn_dec_1 uses VM inv eng 8 on hub 2<o:p></o:p> [ 132.343152] amdgpu 0000:43:00.0: amdgpu: ring vcn_enc_1.0 uses VM inv eng 9 on hub 2<o:p></o:p> [ 132.343657] amdgpu 0000:43:00.0: amdgpu: ring vcn_enc_1.1 uses VM inv eng 10 on hub 2<o:p></o:p> [ 132.344136] amdgpu 0000:43:00.0: amdgpu: ring jpeg_dec_0 uses VM inv eng 11 on hub 2<o:p></o:p> [ 132.344610] amdgpu 0000:43:00.0: amdgpu: ring jpeg_dec_1 uses VM inv eng 12 on hub 2<o:p></o:p> [ 132.378213] amdgpu: Detected AMDGPU 6 Perf Events.<o:p></o:p> [ 132.387349] [drm] Initialized amdgpu 3.46.0 20150101 for 0000:43:00.0 on minor 1<o:p></o:p> [ 132.388530] pcieport 0000:d7:00.0: bridge window [io 0x1000-0x0fff] to [bus d8] add_size 1000<o:p></o:p> [ 132.389078] pcieport 0000:d7:00.0: BAR 13: no space for [io size 0x1000]<o:p></o:p> [ 132.389600] pcieport 0000:d7:00.0: BAR 13: failed to assign [io size 0x1000]<o:p></o:p> [ 132.390091] pcieport 0000:d7:00.0: BAR 13: no space for [io size 0x1000]<o:p></o:p> [ 132.390568] pcieport 0000:d7:00.0: BAR 13: failed to assign [io size 0x1000]<o:p></o:p> [ 155.359200] HelloWorld[3824]: segfault at 68 ip 00007f4c979f764e sp 00007ffc9b3bb610 error 4 in libamdhip64.so.4.4.21432-f9dccde4[7f4c979b3000+2eb000]<o:p></o:p> [ 155.360268] Code: 48 8b 45 e8 64 48 33 04 25 28 00 00 00 74 05 e8 b8 c7 fb ff 48 8b 5d f8 c9 c3 f3 0f 1e fa 55 48 89 e5 48 89 7d f8 48 8b 45 f8 <48> 8b 40 68 5d c3 f3 0f 1e fa 55 48 89 e5 48 89 7d f8 48 8b 45 f8<o:p></o:p> <o:p></o:p> Best regards,<o:p></o:p> Shuotao<o:p></o:p> <o:p></o:p> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in"> From: Andrey Grodzovsky <<a href="mailto:andrey.grodzovsky@amd.com">andrey.grodzovsky@amd.com</a>> Date: Wednesday, April 6, 2022 at 10:36 PM To: Shuotao Xu <<a href="mailto:shuotaoxu@microsoft.com">shuotaoxu@microsoft.com</a>>, <a href="mailto:amd-gfx@lists.freedesktop.org">amd-gfx@lists.freedesktop.org</a> <<a href="mailto:amd-gfx@lists.freedesktop.org">amd-gfx@lists.freedesktop.org</a>> Cc: Ziyue Yang <<a href="mailto:Ziyue.Yang@microsoft.com">Ziyue.Yang@microsoft.com</a>>, Lei Qu <<a href="mailto:Lei.Qu@microsoft.com">Lei.Qu@microsoft.com</a>>, Peng Cheng <<a href="mailto:pengc@microsoft.com">pengc@microsoft.com</a>>, Ran Shu <<a href="mailto:Ran.Shu@microsoft.com">Ran.Shu@microsoft.com</a>> Subject: Re: [EXTERNAL] Re: Code Review Request for AMDGPU Hotplug Support<o:p></o:p> </div> <div> [You don't often get email from <a href="mailto:andrey.grodzovsky@amd.com">andrey.grodzovsky@amd.com</a>. Learn why this is important at <a href="https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Faka.ms%2FLearnAboutSenderIdentification.%255d&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Ca782bea4842e45f15b7408da183d85b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848950411722793%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oY6DZbPB%2BJZSSDdMNg4s44cRc4TnT20Vtew%2Fw4Uziq8%3D&reserved=0"> http://aka.ms/LearnAboutSenderIdentification.]</a> Can you attach dmesg for the failure without your patch against amd-staging-drm-next ? Also, in general, patches for amdgpu upstream branches should be submitted to amd-gfx mailing list inline using git-send which makes it easy to comment and review them inline. Andrey On 2022-04-06 10:25, Shuotao Xu wrote: > Hi Andrey, > > We just tried kernel 5.16 based on > <a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fagd5f%2Flinux.git&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Ca782bea4842e45f15b7408da183d85b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848950411722793%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ssS4iNDEer0H%2FkK1N34xiC76v3TKv4GNYYWvlELU670%3D&reserved=0"> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fagd5f%2Flinux.git&data=05%7C01%7Cshuotaoxu%40microsoft.com%7C93f1fcb8f60541f7b87308da17dae167%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848526184858564%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jz51aMtsG7PIEfLy1jLvLGd%2BsBREvOFf9Gc6BZlmsmU%3D&reserved=0</a> > <<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fagd5f%2Flinux.git&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Ca782bea4842e45f15b7408da183d85b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848950411722793%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ssS4iNDEer0H%2FkK1N34xiC76v3TKv4GNYYWvlELU670%3D&reserved=0">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fagd5f%2Flinux.git&data=05%7C01%7Cshuotaoxu%40microsoft.com%7C93f1fcb8f60541f7b87308da17dae167%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848526184858564%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jz51aMtsG7PIEfLy1jLvLGd%2BsBREvOFf9Gc6BZlmsmU%3D&reserved=0</a>> > amd-staging-drm-next branch, and found out that hotplug did not work out > of box for Rocm compute stack. > > We did not try the rendering stack since we currently are more focused > on AI workloads. > > We have also created a patch against the amd-staging-drm-next branch to > enable hotplug for ROCM stack, which were sent in another later email > with same subject. I am attaching the patch in this email, in case that > you would want to delete that later email. > > Best regards, > > Shuotao > > *From: *Andrey Grodzovsky <<a href="mailto:andrey.grodzovsky@amd.com">andrey.grodzovsky@amd.com</a>> > *Date: *Wednesday, April 6, 2022 at 10:13 PM > *To: *Shuotao Xu <<a href="mailto:shuotaoxu@microsoft.com">shuotaoxu@microsoft.com</a>>, > <a href="mailto:amd-gfx@lists.freedesktop.org">amd-gfx@lists.freedesktop.org</a> <<a href="mailto:amd-gfx@lists.freedesktop.org">amd-gfx@lists.freedesktop.org</a>> > *Cc: *Ziyue Yang <<a href="mailto:Ziyue.Yang@microsoft.com">Ziyue.Yang@microsoft.com</a>>, Lei Qu > <<a href="mailto:Lei.Qu@microsoft.com">Lei.Qu@microsoft.com</a>>, Peng Cheng <<a href="mailto:pengc@microsoft.com">pengc@microsoft.com</a>>, Ran Shu > <<a href="mailto:Ran.Shu@microsoft.com">Ran.Shu@microsoft.com</a>> > *Subject: *[EXTERNAL] Re: Code Review Request for AMDGPU Hotplug Support > > [You don't often get email from <a href="mailto:andrey.grodzovsky@amd.com">andrey.grodzovsky@amd.com</a>. Learn why > this is important at <a href="https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Faka.ms%2FLearnAboutSenderIdentification.%255d&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Ca782bea4842e45f15b7408da183d85b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848950411722793%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oY6DZbPB%2BJZSSDdMNg4s44cRc4TnT20Vtew%2Fw4Uziq8%3D&reserved=0"> http://aka.ms/LearnAboutSenderIdentification.]</a> > <<a href="https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Faka.ms%2FLearnAboutSenderIdentification.%255d&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Ca782bea4842e45f15b7408da183d85b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848950411722793%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oY6DZbPB%2BJZSSDdMNg4s44cRc4TnT20Vtew%2Fw4Uziq8%3D&reserved=0">https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Faka.ms%2FLearnAboutSenderIdentification.%255d&data=05%7C01%7Cshuotaoxu%40microsoft.com%7C93f1fcb8f60541f7b87308da17dae167%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848526184858564%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=HfSwu6SWfoCYyscJqGFdKHBPtaj%2BKB4lyo13zkm6hi4%3D&reserved=0</a>> > > Looks like you are using 5.13 kernel for this work, FYI we added > hot plug support for the graphic stack in 5.14 kernel (see > <a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.phoronix.com%2Fscan.php%3Fpage%3Dnews_item%26px%3DLinux-5.14-AMDGPU-Hot-Unplug&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Ca782bea4842e45f15b7408da183d85b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848950411722793%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2nCzwhL%2Bg8GIJ3WlDacVB6Rbb1hAtHRHhfihZRStzuI%3D&reserved=0"> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.phoronix.com%2Fscan.php%3Fpage%3Dnews_item%26px%3DLinux-5.14-AMDGPU-Hot-Unplug&data=05%7C01%7Cshuotaoxu%40microsoft.com%7C93f1fcb8f60541f7b87308da17dae167%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848526184858564%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4l9mT8zNR%2FvqsEFr7noIDqKf16IGN8xmO2T6jnpipzo%3D&reserved=0)</a> > <<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.phoronix.com%2Fscan.php%3Fpage%3Dnews_item%26px%3DLinux-5.14-AMDGPU-Hot-Unplug&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Ca782bea4842e45f15b7408da183d85b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848950411722793%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2nCzwhL%2Bg8GIJ3WlDacVB6Rbb1hAtHRHhfihZRStzuI%3D&reserved=0">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.phoronix.com%2Fscan.php%3Fpage%3Dnews_item%26px%3DLinux-5.14-AMDGPU-Hot-Unplug&data=05%7C01%7Cshuotaoxu%40microsoft.com%7C93f1fcb8f60541f7b87308da17dae167%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848526184858564%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4l9mT8zNR%2FvqsEFr7noIDqKf16IGN8xmO2T6jnpipzo%3D&reserved=0</a>> > > > I am not sure about the code part since it all touches KFD driver (KFD > team can comment on that) - but I was just wondering if you try 5.14 > kernel would things just work for you out of the box ? > > Andrey > > On 2022-04-05 22:45, Shuotao Xu wrote: >> Dear AMD Colleagues, >> >> We are from Microsoft Research, and are working on GPU disaggregation >> technology. >> >> We have created a new pull requestAdd PCIe hotplug support for amdgpu by >> xushuotao · Pull Request #131 · RadeonOpenCompute/ROCK-Kernel-Driver >> (github.com) >> <<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRadeonOpenCompute%2FROCK-Kernel-Driver%2Fpull%2F131&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Ca782bea4842e45f15b7408da183d85b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848950411722793%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xvW04nplXWAZ7jxsTnABR1JKY7HcHw80nPYJPPpD%2BrI%3D&reserved=0">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRadeonOpenCompute%2FROCK-Kernel-Driver%2Fpull%2F131&data=05%7C01%7Cshuotaoxu%40microsoft.com%7C93f1fcb8f60541f7b87308da17dae167%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848526184858564%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qEeZR6R95jrjAaltd1MnpyFedOiVZaNQuCxcoNog90g%3D&reserved=0 </a>> <<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRadeonOpenCompute%2FROCK-Kernel-Driver%2Fpull%2F131&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Ca782bea4842e45f15b7408da183d85b4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848950411772785%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=EK5BHdupZ2UJsJXTAcYa5rAYoy3uCVvSrTH1uCOS1wQ%3D&reserved=0">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRadeonOpenCompute%2FROCK-Kernel-Driver%2Fpull%2F131&data=05%7C01%7Cshuotaoxu%40microsoft.com%7C93f1fcb8f60541f7b87308da17dae167%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848526184858564%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qEeZR6R95jrjAaltd1MnpyFedOiVZaNQuCxcoNog90g%3D&reserved=0</a>>>in >> ROCK-Kernel-Driver, which will enable PCIe hot-plug support for amdgpu. >> >> We believe the support of hot-plug of GPU devices can open doors for >> many advanced applications in data center in the next few years, and we >> would like to have some reviewers on this PR so we can continue further >> technical discussions around this feature. >> >> Would you please help review this PR? >> >> Thank you very much! >> >> Best regards, >> >> Shuotao Xu >> ><o:p></o:p> </div> </div> </div> </div> </div> </body> </html>