<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""> Please see my inline comments:) <div class=""><br class=""> <div><br class=""> <blockquote type="cite" class=""> <div class="">On Apr 6, 2022, at 11:39 PM, Andrey Grodzovsky <<a href="mailto:andrey.grodzovsky@amd.com" class="">andrey.grodzovsky@amd.com</a>> wrote:</div> <br class="Apple-interchange-newline"> <div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">+ Felix</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">On 2022-04-06 11:11, Shuotao Xu wrote:</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> Hi Andrey,<br class=""> Thanks for your kind comment on linux patch submission protocol, please let me know if there is anyway to rectify it.<br class=""> </blockquote> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Just resend your patch to amd-gfx mailoing list using</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">git-send (see here how to use it -<span class="Apple-converted-space"> </span></span><a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fburzalodowa.wordpress.com%2F2013%2F10%2F05%2Fhow-to-send-patches-with-git-send-email%2F&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sXcPSfZ1VFiizQ%2BWrdvNGDDZhTALFv9im7cNwYsZqPc%3D&reserved=0" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fburzalodowa.wordpress.com%2F2013%2F10%2F05%2Fhow-to-send-patches-with-git-send-email%2F&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sXcPSfZ1VFiizQ%2BWrdvNGDDZhTALFv9im7cNwYsZqPc%3D&reserved=0</a><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">)</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">I suggest adding --cover-letter so you will be able to explain the</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">story behind the patch.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> </div> </blockquote> Yes, thanks for the introduction. We will follow the protocol to resubmit the patch.</div> <div><br class=""> <blockquote type="cite" class=""> <div class=""> <blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> dmesg is fine except with some warning during pci rescan after pci removal of an AMD MI100.<br class=""> The issue is that after this rocm application will return segfault with the amdgpu driver unless the entire amdgpu kernel module is unloaded and loaded, which we did not meet our hotplug requirement. The issues upon investigation are<br class=""> 1) kfd_lock is locked after hotplug, and kfd_open will return fault right away to libhsakmt .<br class=""> </blockquote> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">I see now, kfd_lock is static and so single isntance across all devices</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">and so not going away after device removal but only after driver unload.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">In this case I am not sure it's the best idea to just decrement kfd_lock</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">on device inint since in multi GPU system this might be locked on purpuse because another device is going through reset for example right</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">at this moment.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Felix, kgd2kfd_suspend is called also during device pci remove meaning</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">unblalanced decrement of the lock. Maybe we should not decremnt I</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">adding drm_dev_enter guard in kgd2kfd_suspend to avoid decrment of</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">kfd_locked if we are during PCI remove.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> </div> </blockquote> <div>Yes, I don’t fully understand why kfd_lock’s purpose, and saw the driver never decrements the lock the after Hotplug out.</div> <div>This just a temporary fix by me, and I think AMD should have a better understanding of a correct fix.</div> <div>Ideally, we would like to add and remove gpu nodes flexibly into a bare-metal system, which enables autoscaling a gpu node.</div> <div>This is one of the key features we are working on for GPU disaggregation.</div> <br class=""> <blockquote type="cite" class=""> <div class=""> <blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> 2) iolink/p2plink has anormalies after hotplug, and libhsakmt will found such anomalies and return error.<br class=""> </blockquote> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Can you point to what abnormalities ? Part of PCI hot unplug we clean</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">all sysfs files and this looks like part of it, do you see sysfs file</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">already exsist error on next pci_rescan ?</span><br class=""> </div> </blockquote> The following demonstrates the anomalies:</div> <div>We have three kfd nodes in the systems</div> <div>0: cpu</div> <div>1: cpu</div> <div>2: amdgpu</div> <div><br class=""> </div> <div>--Before hotplug out: there is a io-link between 0 and 2</div> <div>$cat /sys/devices/virtual/kfd/kfd/topology/nodes/2/io_links/0/properties</div> <div>type 2</div> <div>version_major 0</div> <div>version_minor 0</div> <div>node_from 2</div> <div>node_to 0</div> <div>weight 20</div> <div>min_latency 0</div> <div>max_latency 0</div> <div>min_bandwidth 312</div> <div>max_bandwidth 8000</div> <div>recommended_transfer_size 0</div> <div>flags 1</div> <div><br class=""> </div> <div>--After hotplug out: </div> <div>Anomaly 0: this io-link between 0 and 2 should have been removed, which is not</div> <div> <div>$ cat /sys/devices/virtual/kfd/kfd/topology/nodes/0/io_links/1/properties</div> <div>type 2</div> <div>version_major 0</div> <div>version_minor 0</div> <div>node_from 0</div> <div>node_to 2</div> <div>weight 20</div> <div>min_latency 0</div> <div>max_latency 0</div> <div>min_bandwidth 312</div> <div>max_bandwidth 8000</div> <div>recommended_transfer_size 0</div> <div>flags 3</div> <div><br class=""> </div> <div>—After hotplug in: </div> <div>Anomaly 1: another io-link between 0 and 2 is created.</div> <div>Anomaly 2: the node id of 2 is 3, which should have been 3, although libhsakmt won’t pick this error up.</div> <div> This is because of atomic counter, topology_crat_proximity_domain as mentioned by Joshi later.</div> <div> <div>$ cat /sys/devices/virtual/kfd/kfd/topology/nodes/0/io_links/2/properties</div> <div>type 2</div> <div>version_major 0</div> <div>version_minor 0</div> <div>node_from 0</div> <div>node_to 3</div> <div>weight 20</div> <div>min_latency 0</div> <div>max_latency 0</div> <div>min_bandwidth 312</div> <div>max_bandwidth 8000</div> <div>recommended_transfer_size 0</div> <div>flags 3</div> </div> <div><br class=""> </div> <div>Anomaly 3: node_from should have been 2 instead of 3. </div> <div> libhsakmt will pick this anomaly and return an error when taking the topology snapshot. </div> <div><span class="">$ cat /sys/devices/virtual/kfd/kfd/topology/nodes/2/io_links/0/properties</span></div> </div> <div> <div>type 2</div> <div>version_major 0</div> <div>version_minor 0</div> <div>node_from 3</div> <div>node_to 0</div> <div>weight 20</div> <div>min_latency 0</div> <div>max_latency 0</div> <div>min_bandwidth 312</div> <div>max_bandwidth 8000</div> <div>recommended_transfer_size 0</div> <div>flags 1</div> <div><br class=""> </div> Our patch at least have fixed those three anomalies in the kfd topology in a single-GPU system.</div> <div><br class=""> </div> <div> <blockquote type="cite" class=""> <div class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> Our patch has been tested with a single-instance AMD MI100 GPU and showed it worked.<br class=""> </blockquote> <br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Exactly, for multi GPU system arbitrary decrementing kfd_lock on device load</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">can be problematic.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> </div> </blockquote> I was hoping AMD would do the correct fix; the driver code is quite convoluted from an outsider.</div> <div>Reverse-engineering without much documentation has been quite painful for me.</div> <div><br class=""> </div> <div> <blockquote type="cite" class=""> <div class=""><br class=""> </div> </blockquote> <blockquote type="cite" class=""> <div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Andrey</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> </div> </blockquote> <div>Thank you very much for prompt discussion! We really appreciate it!</div> <div><br class=""> </div> <div>Best regards,</div> <div>Shuotao</div> <div><br class=""> </div> <blockquote type="cite" class=""> <div class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> <blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""> I am attaching the dmesg after rescan anyway, which will show the warning and the segfault.<br class=""> [ 132.054822] pci 0000:43:00.0: [1002:738c] type 00 class 0x038000<br class=""> [ 132.054856] pci 0000:43:00.0: reg 0x10: [mem 0x38b000000000-0x38b7ffffffff 64bit pref]<br class=""> [ 132.054877] pci 0000:43:00.0: reg 0x18: [mem 0x38b800000000-0x38b8001fffff 64bit pref]<br class=""> [ 132.054890] pci 0000:43:00.0: reg 0x20: [io 0xa000-0xa0ff]<br class=""> [ 132.054904] pci 0000:43:00.0: reg 0x24: [mem 0xb8400000-0xb847ffff]<br class=""> [ 132.054918] pci 0000:43:00.0: reg 0x30: [mem 0xb8480000-0xb849ffff pref]<br class=""> [ 132.055134] pci 0000:43:00.0: PME# supported from D1 D2 D3hot D3cold<br class=""> [ 132.055217] pci 0000:43:00.0: 63.008 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x8 link at 0000:3c:14.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)<br class=""> [ 132.056001] pci 0000:43:00.0: Adding to iommu group 73<br class=""> [ 132.057943] pci 0000:43:00.0: BAR 0: assigned [mem 0x38b000000000-0x38b7ffffffff 64bit pref]<br class=""> [ 132.057960] pci 0000:43:00.0: BAR 2: assigned [mem 0x38b800000000-0x38b8001fffff 64bit pref]<br class=""> [ 132.057974] pci 0000:43:00.0: BAR 5: assigned [mem 0xb8400000-0xb847ffff]<br class=""> [ 132.057981] pci 0000:43:00.0: BAR 6: assigned [mem 0xb8480000-0xb849ffff pref]<br class=""> [ 132.057984] pci 0000:43:00.0: BAR 4: assigned [io 0xa000-0xa0ff]<br class=""> [ 132.058429] ======================================================<br class=""> [ 132.058453] WARNING: possible circular locking dependency detected<br class=""> [ 132.058477] 5.16.0-kfd+ #1 Not tainted<br class=""> [ 132.058492] ------------------------------------------------------<br class=""> [ 132.058515] bash/3632 is trying to acquire lock:<br class=""> [ 132.058534] ffffadee20adfb50 ((work_completion)(&wfc.work)){+.+.}-{0:0}, at: __flush_work+0x2f5/0x470<br class=""> [ 132.058554] [drm] initializing kernel modesetting (ARCTURUS 0x1002:0x738C 0x1002:0x0C34 0x01).<br class=""> [ 132.058577]<br class=""> <span class="Apple-converted-space"> </span>but task is already holding lock:<br class=""> [ 132.058580] ffffffffa3c62308<br class=""> [ 132.058630] amdgpu 0000:43:00.0: amdgpu: Trusted Memory Zone (TMZ) feature not supported<br class=""> [ 132.058638] (<br class=""> [ 132.058678] [drm] register mmio base: 0xB8400000<br class=""> [ 132.058683] pci_rescan_remove_lock<br class=""> [ 132.058694] [drm] register mmio size: 524288<br class=""> [ 132.058713] ){+.+.}-{3:3}, at: rescan_store+0x55/0x90<br class=""> [ 132.058773]<br class=""> <span class="Apple-converted-space"> </span>which lock already depends on the new lock.<br class=""> [ 132.058804]<br class=""> <span class="Apple-converted-space"> </span>the existing dependency chain (in reverse order) is:<br class=""> [ 132.058819] [drm] add ip block number 0 <soc15_common><br class=""> [ 132.058831]<br class=""> <span class="Apple-converted-space"> </span>-> #1 (<br class=""> [ 132.058854] [drm] add ip block number 1 <gmc_v9_0><br class=""> [ 132.058858] [drm] add ip block number 2 <vega20_ih><br class=""> [ 132.058874] pci_rescan_remove_lock<br class=""> [ 132.058894] [drm] add ip block number 3 <psp><br class=""> [ 132.058915] ){+.+.}-{3:3}<br class=""> [ 132.058931] [drm] add ip block number 4 <smu><br class=""> [ 132.058951] :<br class=""> [ 132.058965] [drm] add ip block number 5 <gfx_v9_0><br class=""> [ 132.058986] __mutex_lock+0xa4/0x990<br class=""> [ 132.058996] [drm] add ip block number 6 <sdma_v4_0><br class=""> [ 132.059016] i801_add_tco_spt.isra.20+0x2a/0x1a0<br class=""> [ 132.059033] [drm] add ip block number 7 <vcn_v2_5><br class=""> [ 132.059054] i801_add_tco+0xf6/0x110<br class=""> [ 132.059075] [drm] add ip block number 8 <jpeg_v2_5><br class=""> [ 132.059096] i801_probe+0x402/0x860<br class=""> [ 132.059151] local_pci_probe+0x40/0x90<br class=""> [ 132.059170] work_for_cpu_fn+0x10/0x20<br class=""> [ 132.059189] process_one_work+0x2a4/0x640<br class=""> [ 132.059208] worker_thread+0x228/0x3f0<br class=""> [ 132.059227] kthread+0x16d/0x1a0<br class=""> [ 132.059795] ret_from_fork+0x1f/0x30<br class=""> [ 132.060337]<br class=""> <span class="Apple-converted-space"> </span>-> #0 ((work_completion)(&wfc.work)){+.+.}-{0:0}:<br class=""> [ 132.061405] __lock_acquire+0x1552/0x1ac0<br class=""> [ 132.061950] lock_acquire+0x26c/0x300<br class=""> [ 132.062484] __flush_work+0x315/0x470<br class=""> [ 132.063009] work_on_cpu+0x98/0xc0<br class=""> [ 132.063526] pci_device_probe+0x1bc/0x1d0<br class=""> [ 132.064036] really_probe+0x102/0x450<br class=""> [ 132.064532] __driver_probe_device+0x100/0x170<br class=""> [ 132.065020] driver_probe_device+0x1f/0xa0<br class=""> [ 132.065497] __device_attach_driver+0x6b/0xe0<br class=""> [ 132.065975] bus_for_each_drv+0x6a/0xb0<br class=""> [ 132.066449] __device_attach+0xe2/0x160<br class=""> [ 132.066912] pci_bus_add_device+0x4a/0x80<br class=""> [ 132.067365] pci_bus_add_devices+0x2c/0x70<br class=""> [ 132.067812] pci_bus_add_devices+0x65/0x70<br class=""> [ 132.068253] pci_bus_add_devices+0x65/0x70<br class=""> [ 132.068688] pci_bus_add_devices+0x65/0x70<br class=""> [ 132.068936] amdgpu 0000:43:00.0: amdgpu: Fetched VBIOS from ROM BAR<br class=""> [ 132.069109] pci_bus_add_devices+0x65/0x70<br class=""> [ 132.069602] amdgpu: ATOM BIOS: 113-D3431401-X00<br class=""> [ 132.070058] pci_bus_add_devices+0x65/0x70<br class=""> [ 132.070572] [drm] VCN(0) decode is enabled in VM mode<br class=""> [ 132.070997] pci_rescan_bus+0x23/0x30<br class=""> [ 132.071000] rescan_store+0x61/0x90<br class=""> [ 132.071003] kernfs_fop_write_iter+0x132/0x1b0<br class=""> [ 132.071501] [drm] VCN(1) decode is enabled in VM mode<br class=""> [ 132.071964] new_sync_write+0x11f/0x1b0<br class=""> [ 132.072432] [drm] VCN(0) encode is enabled in VM mode<br class=""> [ 132.072900] vfs_write+0x35b/0x3b0<br class=""> [ 132.073376] [drm] VCN(1) encode is enabled in VM mode<br class=""> [ 132.073847] ksys_write+0xa7/0xe0<br class=""> [ 132.074335] [drm] JPEG(0) JPEG decode is enabled in VM mode<br class=""> [ 132.074803] do_syscall_64+0x34/0x80<br class=""> [ 132.074808] entry_SYSCALL_64_after_hwframe+0x44/0xae<br class=""> [ 132.074811]<br class=""> <span class="Apple-converted-space"> </span>other info that might help us debug this:<br class=""> [ 132.074813] Possible unsafe locking scenario:<br class=""> [ 132.075302] [drm] JPEG(1) JPEG decode is enabled in VM mode<br class=""> [ 132.075779] CPU0 CPU1<br class=""> [ 132.076361] amdgpu 0000:43:00.0: amdgpu: MEM ECC is active.<br class=""> [ 132.076765] ---- ----<br class=""> [ 132.077265] amdgpu 0000:43:00.0: amdgpu: SRAM ECC is active.<br class=""> [ 132.078649] lock(pci_rescan_remove_lock);<br class=""> [ 132.078652] lock((work_completion)(&wfc.work));<br class=""> [ 132.078653] lock(pci_rescan_remove_lock);<br class=""> [ 132.078655] lock((work_completion)(&wfc.work));<br class=""> [ 132.078656]<br class=""> <span class="Apple-converted-space"> </span>*** DEADLOCK ***<br class=""> [ 132.078656] 5 locks held by bash/3632:<br class=""> [ 132.078658] #0: ffff9c39c7b89438<br class=""> [ 132.079612] amdgpu 0000:43:00.0: amdgpu: RAS INFO: ras initialized successfully, hardware ability[7fff] ras_mask[7fff]<br class=""> [ 132.080089] (<br class=""> [ 132.080602] [drm] vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit<br class=""> [ 132.081082] sb_writers<br class=""> [ 132.081601] amdgpu 0000:43:00.0: amdgpu: VRAM: 32752M 0x0000008000000000 - 0x00000087FEFFFFFF (32752M used)<br class=""> [ 132.082102] #6<br class=""> [ 132.082630] amdgpu 0000:43:00.0: amdgpu: GART: 512M 0x0000000000000000 - 0x000000001FFFFFFF<br class=""> [ 132.083152] ){.+.+}-{0:0}<br class=""> [ 132.083687] amdgpu 0000:43:00.0: amdgpu: AGP: 267878400M 0x0000008800000000 - 0x0000FFFFFFFFFFFF<br class=""> [ 132.084210] , at: ksys_write+0xa7/0xe0<br class=""> [ 132.085708] [drm] Detected VRAM RAM=32752M, BAR=32768M<br class=""> [ 132.086205] #1:<br class=""> [ 132.086733] [drm] RAM width 4096bits HBM<br class=""> [ 132.087269] ffff9c5959011088<br class=""> [ 132.087890] [drm] amdgpu: 32752M of VRAM memory ready<br class=""> [ 132.088389] (<br class=""> [ 132.088972] [drm] amdgpu: 32752M of GTT memory ready.<br class=""> [ 132.089572] &of->mutex<br class=""> [ 132.090206] [drm] GART: num cpu pages 131072, num gpu pages 131072<br class=""> [ 132.090804] ){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x103/0x1b0<br class=""> [ 132.090808] #2: ffff9c39c882a9e0 (kn->active#423){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x10c/0x1b0<br class=""> [ 132.091639] [drm] PCIE GART of 512M enabled.<br class=""> [ 132.092117] #3:<br class=""> [ 132.092801] [drm] PTB located at 0x0000008000000000<br class=""> [ 132.093480] ffffffffa3c62308<br class=""> [ 132.094566] amdgpu 0000:43:00.0: amdgpu: PSP runtime database doesn't exist<br class=""> [ 132.094822] (pci_rescan_remove_lock){+.+.}-{3:3}, at: rescan_store+0x55/0x90<br class=""> [ 132.094827] #4: ffff9c597392b248 (&dev->mutex){....}-{3:3}, at: __device_attach+0x39/0x160<br class=""> [ 132.094835]<br class=""> <span class="Apple-converted-space"> </span>stack backtrace:<br class=""> [ 132.097098] [drm] Found VCN firmware Version ENC: 1.1 DEC: 1 VEP: 0 Revision: 21<br class=""> [ 132.097467] CPU: 47 PID: 3632 Comm: bash Not tainted 5.16.0-kfd+ #1<br class=""> [ 132.098169] amdgpu 0000:43:00.0: amdgpu: Will use PSP to load VCN firmware<br class=""> [ 132.098839] Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 2.1 08/14/2018<br class=""> [ 132.098841] Call Trace:<br class=""> [ 132.098842] <TASK><br class=""> [ 132.098843] dump_stack_lvl+0x44/0x57<br class=""> [ 132.098848] check_noncircular+0x105/0x120<br class=""> [ 132.098853] ? unwind_get_return_address+0x1b/0x30<br class=""> [ 132.112924] ? register_lock_class+0x46/0x780<br class=""> [ 132.113630] ? __lock_acquire+0x1552/0x1ac0<br class=""> [ 132.114342] __lock_acquire+0x1552/0x1ac0<br class=""> [ 132.115050] lock_acquire+0x26c/0x300<br class=""> [ 132.115755] ? __flush_work+0x2f5/0x470<br class=""> [ 132.116460] ? lock_is_held_type+0xdf/0x130<br class=""> [ 132.117177] __flush_work+0x315/0x470<br class=""> [ 132.117890] ? __flush_work+0x2f5/0x470<br class=""> [ 132.118604] ? lock_is_held_type+0xdf/0x130<br class=""> [ 132.119305] ? mark_held_locks+0x49/0x70<br class=""> [ 132.119981] ? queue_work_on+0x2f/0x70<br class=""> [ 132.120645] ? lockdep_hardirqs_on+0x79/0x100<br class=""> [ 132.121300] work_on_cpu+0x98/0xc0<br class=""> [ 132.121702] [drm] reserve 0x400000 from 0x87fec00000 for PSP TMR<br class=""> [ 132.121947] ? __traceiter_workqueue_execute_end+0x40/0x40<br class=""> [ 132.123270] ? pci_device_shutdown+0x60/0x60<br class=""> [ 132.123880] pci_device_probe+0x1bc/0x1d0<br class=""> [ 132.124475] really_probe+0x102/0x450<br class=""> [ 132.125060] __driver_probe_device+0x100/0x170<br class=""> [ 132.125641] driver_probe_device+0x1f/0xa0<br class=""> [ 132.126215] __device_attach_driver+0x6b/0xe0<br class=""> [ 132.126797] ? driver_allows_async_probing+0x50/0x50<br class=""> [ 132.127383] ? driver_allows_async_probing+0x50/0x50<br class=""> [ 132.127960] bus_for_each_drv+0x6a/0xb0<br class=""> [ 132.128528] __device_attach+0xe2/0x160<br class=""> [ 132.129095] pci_bus_add_device+0x4a/0x80<br class=""> [ 132.129659] pci_bus_add_devices+0x2c/0x70<br class=""> [ 132.130213] pci_bus_add_devices+0x65/0x70<br class=""> [ 132.130753] pci_bus_add_devices+0x65/0x70<br class=""> [ 132.131283] pci_bus_add_devices+0x65/0x70<br class=""> [ 132.131780] pci_bus_add_devices+0x65/0x70<br class=""> [ 132.132270] pci_bus_add_devices+0x65/0x70<br class=""> [ 132.132757] pci_rescan_bus+0x23/0x30<br class=""> [ 132.133233] rescan_store+0x61/0x90<br class=""> [ 132.133701] kernfs_fop_write_iter+0x132/0x1b0<br class=""> [ 132.134167] new_sync_write+0x11f/0x1b0<br class=""> [ 132.134627] vfs_write+0x35b/0x3b0<br class=""> [ 132.135062] ksys_write+0xa7/0xe0<br class=""> [ 132.135503] do_syscall_64+0x34/0x80<br class=""> [ 132.135933] entry_SYSCALL_64_after_hwframe+0x44/0xae<br class=""> [ 132.136358] RIP: 0033:0x7f0058a73224<br class=""> [ 132.136775] Code: 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 8d 05 c1 07 2e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 f3 c3 66 90 41 54 55 49 89 d4 53 48 89 f5<br class=""> [ 132.137663] RSP: 002b:00007ffc4f4c71a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001<br class=""> [ 132.138121] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f0058a73224<br class=""> [ 132.138590] RDX: 0000000000000002 RSI: 000055d466c24450 RDI: 0000000000000001<br class=""> [ 132.139064] RBP: 000055d466c24450 R08: 000000000000000a R09: 0000000000000001<br class=""> [ 132.139532] R10: 000000000000000a R11: 0000000000000246 R12: 00007f0058d4f760<br class=""> [ 132.140003] R13: 0000000000000002 R14: 00007f0058d4b2a0 R15: 00007f0058d4a760<br class=""> [ 132.140485] </TASK><br class=""> [ 132.183669] amdgpu 0000:43:00.0: amdgpu: HDCP: optional hdcp ta ucode is not available<br class=""> [ 132.184214] amdgpu 0000:43:00.0: amdgpu: DTM: optional dtm ta ucode is not available<br class=""> [ 132.184735] amdgpu 0000:43:00.0: amdgpu: RAP: optional rap ta ucode is not available<br class=""> [ 132.185234] amdgpu 0000:43:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available<br class=""> [ 132.185823] amdgpu 0000:43:00.0: amdgpu: use vbios provided pptable<br class=""> [ 132.186327] amdgpu 0000:43:00.0: amdgpu: smc_dpm_info table revision(format.content): 4.6<br class=""> [ 132.188783] amdgpu 0000:43:00.0: amdgpu: SMU is initialized successfully!<br class=""> [ 132.190039] [drm] kiq ring mec 2 pipe 1 q 0<br class=""> [ 132.203608] [drm] VCN decode and encode initialized successfully(under DPG Mode).<br class=""> [ 132.204178] [drm] JPEG decode initialized successfully.<br class=""> [ 132.246079] kfd kfd: amdgpu: Allocated 3969056 bytes on gart<br class=""> [ 132.327589] memmap_init_zone_device initialised 8388608 pages in 64ms<br class=""> [ 132.328139] amdgpu: HMM registered 32752MB device memory<br class=""> [ 132.328784] amdgpu: Virtual CRAT table created for GPU<br class=""> [ 132.329844] amdgpu: Topology: Add dGPU node [0x738c:0x1002]<br class=""> [ 132.330387] kfd kfd: amdgpu: added device 1002:738c<br class=""> [ 132.330965] amdgpu 0000:43:00.0: amdgpu: SE 8, SH per SE 1, CU per SH 16, active_cu_number 72<br class=""> [ 132.331725] amdgpu 0000:43:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 0 on hub 0<br class=""> [ 132.332296] amdgpu 0000:43:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 1 on hub 0<br class=""> [ 132.332856] amdgpu 0000:43:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 4 on hub 0<br class=""> [ 132.333414] amdgpu 0000:43:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 5 on hub 0<br class=""> [ 132.333965] amdgpu 0000:43:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 6 on hub 0<br class=""> [ 132.334507] amdgpu 0000:43:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 7 on hub 0<br class=""> [ 132.335057] amdgpu 0000:43:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 8 on hub 0<br class=""> [ 132.335594] amdgpu 0000:43:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 9 on hub 0<br class=""> [ 132.336137] amdgpu 0000:43:00.0: amdgpu: ring kiq_2.1.0 uses VM inv eng 10 on hub 0<br class=""> [ 132.336679] amdgpu 0000:43:00.0: amdgpu: ring sdma0 uses VM inv eng 0 on hub 1<br class=""> [ 132.337234] amdgpu 0000:43:00.0: amdgpu: ring sdma1 uses VM inv eng 1 on hub 1<br class=""> [ 132.337790] amdgpu 0000:43:00.0: amdgpu: ring sdma2 uses VM inv eng 4 on hub 1<br class=""> [ 132.338343] amdgpu 0000:43:00.0: amdgpu: ring sdma3 uses VM inv eng 5 on hub 1<br class=""> [ 132.338906] amdgpu 0000:43:00.0: amdgpu: ring sdma4 uses VM inv eng 6 on hub 1<br class=""> [ 132.339448] amdgpu 0000:43:00.0: amdgpu: ring sdma5 uses VM inv eng 0 on hub 2<br class=""> [ 132.339987] amdgpu 0000:43:00.0: amdgpu: ring sdma6 uses VM inv eng 1 on hub 2<br class=""> [ 132.340519] amdgpu 0000:43:00.0: amdgpu: ring sdma7 uses VM inv eng 4 on hub 2<br class=""> [ 132.341041] amdgpu 0000:43:00.0: amdgpu: ring vcn_dec_0 uses VM inv eng 5 on hub 2<br class=""> [ 132.341570] amdgpu 0000:43:00.0: amdgpu: ring vcn_enc_0.0 uses VM inv eng 6 on hub 2<br class=""> [ 132.342101] amdgpu 0000:43:00.0: amdgpu: ring vcn_enc_0.1 uses VM inv eng 7 on hub 2<br class=""> [ 132.342630] amdgpu 0000:43:00.0: amdgpu: ring vcn_dec_1 uses VM inv eng 8 on hub 2<br class=""> [ 132.343152] amdgpu 0000:43:00.0: amdgpu: ring vcn_enc_1.0 uses VM inv eng 9 on hub 2<br class=""> [ 132.343657] amdgpu 0000:43:00.0: amdgpu: ring vcn_enc_1.1 uses VM inv eng 10 on hub 2<br class=""> [ 132.344136] amdgpu 0000:43:00.0: amdgpu: ring jpeg_dec_0 uses VM inv eng 11 on hub 2<br class=""> [ 132.344610] amdgpu 0000:43:00.0: amdgpu: ring jpeg_dec_1 uses VM inv eng 12 on hub 2<br class=""> [ 132.378213] amdgpu: Detected AMDGPU 6 Perf Events.<br class=""> [ 132.387349] [drm] Initialized amdgpu 3.46.0 20150101 for 0000:43:00.0 on minor 1<br class=""> [ 132.388530] pcieport 0000:d7:00.0: bridge window [io 0x1000-0x0fff] to [bus d8] add_size 1000<br class=""> [ 132.389078] pcieport 0000:d7:00.0: BAR 13: no space for [io size 0x1000]<br class=""> [ 132.389600] pcieport 0000:d7:00.0: BAR 13: failed to assign [io size 0x1000]<br class=""> [ 132.390091] pcieport 0000:d7:00.0: BAR 13: no space for [io size 0x1000]<br class=""> [ 132.390568] pcieport 0000:d7:00.0: BAR 13: failed to assign [io size 0x1000]<br class=""> [ 155.359200] HelloWorld[3824]: segfault at 68 ip 00007f4c979f764e sp 00007ffc9b3bb610 error 4 in libamdhip64.so.4.4.21432-f9dccde4[7f4c979b3000+2eb000]<br class=""> [ 155.360268] Code: 48 8b 45 e8 64 48 33 04 25 28 00 00 00 74 05 e8 b8 c7 fb ff 48 8b 5d f8 c9 c3 f3 0f 1e fa 55 48 89 e5 48 89 7d f8 48 8b 45 f8 <48> 8b 40 68 5d c3 f3 0f 1e fa 55 48 89 e5 48 89 7d f8 48 8b 45 f8<br class=""> Best regards,<br class=""> Shuotao<br class=""> *From: *Andrey Grodzovsky <<a href="mailto:andrey.grodzovsky@amd.com" class="">andrey.grodzovsky@amd.com</a>><br class=""> *Date: *Wednesday, April 6, 2022 at 10:36 PM<br class=""> *To: *Shuotao Xu <<a href="mailto:shuotaoxu@microsoft.com" class="">shuotaoxu@microsoft.com</a>>, <a href="mailto:amd-gfx@lists.freedesktop.org" class="">amd-gfx@lists.freedesktop.org</a> <<a href="mailto:amd-gfx@lists.freedesktop.org" class="">amd-gfx@lists.freedesktop.org</a>><br class=""> *Cc: *Ziyue Yang <<a href="mailto:Ziyue.Yang@microsoft.com" class="">Ziyue.Yang@microsoft.com</a>>, Lei Qu <<a href="mailto:Lei.Qu@microsoft.com" class="">Lei.Qu@microsoft.com</a>>, Peng Cheng <<a href="mailto:pengc@microsoft.com" class="">pengc@microsoft.com</a>>, Ran Shu <<a href="mailto:Ran.Shu@microsoft.com" class="">Ran.Shu@microsoft.com</a>><br class=""> *Subject: *Re: [EXTERNAL] Re: Code Review Request for AMDGPU Hotplug Support<br class=""> [You don't often get email from<span class="Apple-converted-space"> </span><a href="mailto:andrey.grodzovsky@amd.com" class="">andrey.grodzovsky@amd.com</a>. Learn why this is important at<span class="Apple-converted-space"> </span><a href="http://aka.ms/LearnAboutSenderIdentification" class="">http://aka.ms/LearnAboutSenderIdentification</a>.] <<a href="https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Faka.ms%2FLearnAboutSenderIdentification.%255d&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0IDdt8tcUR%2BZ7fU0OPlCsICmLNtMgBYF2I6Tnh4rNLE%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Faka.ms%2FLearnAboutSenderIdentification.%255d&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0IDdt8tcUR%2BZ7fU0OPlCsICmLNtMgBYF2I6Tnh4rNLE%3D&reserved=0</a>><br class=""> Can you attach dmesg for the failure without your patch against<br class=""> amd-staging-drm-next ?<br class=""> Also, in general, patches for amdgpu upstream branches should be<br class=""> submitted to amd-gfx mailing list inline using git-send which makes it<br class=""> easy to comment and review them inline.<br class=""> Andrey<br class=""> On 2022-04-06 10:25, Shuotao Xu wrote:<br class=""> <blockquote type="cite" class="">Hi Andrey,<br class=""> <br class=""> We just tried kernel 5.16 based on<br class=""> <a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fagd5f%2Flinux.git&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bQPzShlHhTX67qtY1HJ6gWKVn0Nr2YawHZ5K1EQMi%2F0%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fagd5f%2Flinux.git&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bQPzShlHhTX67qtY1HJ6gWKVn0Nr2YawHZ5K1EQMi%2F0%3D&reserved=0</a><span class="Apple-converted-space"> </span><br class=""> </blockquote> <<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fagd5f%2Flinux.git&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bQPzShlHhTX67qtY1HJ6gWKVn0Nr2YawHZ5K1EQMi%2F0%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fagd5f%2Flinux.git&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bQPzShlHhTX67qtY1HJ6gWKVn0Nr2YawHZ5K1EQMi%2F0%3D&reserved=0</a>><br class=""> <blockquote type="cite" class=""><<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fagd5f%2Flinux.git&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bQPzShlHhTX67qtY1HJ6gWKVn0Nr2YawHZ5K1EQMi%2F0%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fagd5f%2Flinux.git&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bQPzShlHhTX67qtY1HJ6gWKVn0Nr2YawHZ5K1EQMi%2F0%3D&reserved=0</a><span class="Apple-converted-space"> </span><br class=""> </blockquote> <<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fagd5f%2Flinux.git&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bQPzShlHhTX67qtY1HJ6gWKVn0Nr2YawHZ5K1EQMi%2F0%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.freedesktop.org%2Fagd5f%2Flinux.git&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bQPzShlHhTX67qtY1HJ6gWKVn0Nr2YawHZ5K1EQMi%2F0%3D&reserved=0</a>>><br class=""> <blockquote type="cite" class="">amd-staging-drm-next branch, and found out that hotplug did not work out<br class=""> of box for Rocm compute stack.<br class=""> <br class=""> We did not try the rendering stack since we currently are more focused<br class=""> on AI workloads.<br class=""> <br class=""> We have also created a patch against the amd-staging-drm-next branch to<br class=""> enable hotplug for ROCM stack, which were sent in another later email<br class=""> with same subject. I am attaching the patch in this email, in case that<br class=""> you would want to delete that later email.<br class=""> <br class=""> Best regards,<br class=""> <br class=""> Shuotao<br class=""> <br class=""> *From: *Andrey Grodzovsky <<a href="mailto:andrey.grodzovsky@amd.com" class="">andrey.grodzovsky@amd.com</a>><br class=""> *Date: *Wednesday, April 6, 2022 at 10:13 PM<br class=""> *To: *Shuotao Xu <<a href="mailto:shuotaoxu@microsoft.com" class="">shuotaoxu@microsoft.com</a>>,<br class=""> <a href="mailto:amd-gfx@lists.freedesktop.org" class="">amd-gfx@lists.freedesktop.org</a><span class="Apple-converted-space"> </span><<a href="mailto:amd-gfx@lists.freedesktop.org" class="">amd-gfx@lists.freedesktop.org</a>><br class=""> *Cc: *Ziyue Yang <<a href="mailto:Ziyue.Yang@microsoft.com" class="">Ziyue.Yang@microsoft.com</a>>, Lei Qu<br class=""> <<a href="mailto:Lei.Qu@microsoft.com" class="">Lei.Qu@microsoft.com</a>>, Peng Cheng <<a href="mailto:pengc@microsoft.com" class="">pengc@microsoft.com</a>>, Ran Shu<br class=""> <<a href="mailto:Ran.Shu@microsoft.com" class="">Ran.Shu@microsoft.com</a>><br class=""> *Subject: *[EXTERNAL] Re: Code Review Request for AMDGPU Hotplug Support<br class=""> <br class=""> [You don't often get email from<span class="Apple-converted-space"> </span><a href="mailto:andrey.grodzovsky@amd.com" class="">andrey.grodzovsky@amd.com</a>. Learn why<br class=""> this is important at<span class="Apple-converted-space"> </span><a href="http://aka.ms/LearnAboutSenderIdentification" class="">http://aka.ms/LearnAboutSenderIdentification</a>.]<span class="Apple-converted-space"> </span><br class=""> </blockquote> <<a href="https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Faka.ms%2FLearnAboutSenderIdentification.%255d&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0IDdt8tcUR%2BZ7fU0OPlCsICmLNtMgBYF2I6Tnh4rNLE%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Faka.ms%2FLearnAboutSenderIdentification.%255d&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0IDdt8tcUR%2BZ7fU0OPlCsICmLNtMgBYF2I6Tnh4rNLE%3D&reserved=0</a>><br class=""> <blockquote type="cite" class=""><<a href="https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Faka.ms%2FLearnAboutSenderIdentification.%255d&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0IDdt8tcUR%2BZ7fU0OPlCsICmLNtMgBYF2I6Tnh4rNLE%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Faka.ms%2FLearnAboutSenderIdentification.%255d&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0IDdt8tcUR%2BZ7fU0OPlCsICmLNtMgBYF2I6Tnh4rNLE%3D&reserved=0</a><span class="Apple-converted-space"> </span><br class=""> </blockquote> <<a href="https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Faka.ms%2FLearnAboutSenderIdentification.%255d&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0IDdt8tcUR%2BZ7fU0OPlCsICmLNtMgBYF2I6Tnh4rNLE%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Faka.ms%2FLearnAboutSenderIdentification.%255d&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0IDdt8tcUR%2BZ7fU0OPlCsICmLNtMgBYF2I6Tnh4rNLE%3D&reserved=0</a>>><br class=""> <blockquote type="cite" class=""><br class=""> Looks like you are using 5.13 kernel for this work, FYI we added<br class=""> hot plug support for the graphic stack in 5.14 kernel (see<br class=""> <a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.phoronix.com%2Fscan.php%3Fpage%3Dnews_item%26px%3DLinux-5.14-AMDGPU-Hot-Unplug&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u8%2FSr%2BF8k9qZDO8%2F%2Fg3Eti0OaeJgM5fVPAaw2N7Bf6I%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.phoronix.com%2Fscan.php%3Fpage%3Dnews_item%26px%3DLinux-5.14-AMDGPU-Hot-Unplug&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u8%2FSr%2BF8k9qZDO8%2F%2Fg3Eti0OaeJgM5fVPAaw2N7Bf6I%3D&reserved=0</a>)<span class="Apple-converted-space"> </span><br class=""> </blockquote> <<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.phoronix.com%2Fscan.php%3Fpage%3Dnews_item%26px%3DLinux-5.14-AMDGPU-Hot-Unplug&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u8%2FSr%2BF8k9qZDO8%2F%2Fg3Eti0OaeJgM5fVPAaw2N7Bf6I%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.phoronix.com%2Fscan.php%3Fpage%3Dnews_item%26px%3DLinux-5.14-AMDGPU-Hot-Unplug&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u8%2FSr%2BF8k9qZDO8%2F%2Fg3Eti0OaeJgM5fVPAaw2N7Bf6I%3D&reserved=0</a>><br class=""> <blockquote type="cite" class=""><<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.phoronix.com%2Fscan.php%3Fpage%3Dnews_item%26px%3DLinux-5.14-AMDGPU-Hot-Unplug&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u8%2FSr%2BF8k9qZDO8%2F%2Fg3Eti0OaeJgM5fVPAaw2N7Bf6I%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.phoronix.com%2Fscan.php%3Fpage%3Dnews_item%26px%3DLinux-5.14-AMDGPU-Hot-Unplug&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u8%2FSr%2BF8k9qZDO8%2F%2Fg3Eti0OaeJgM5fVPAaw2N7Bf6I%3D&reserved=0</a><span class="Apple-converted-space"> </span><br class=""> </blockquote> <<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.phoronix.com%2Fscan.php%3Fpage%3Dnews_item%26px%3DLinux-5.14-AMDGPU-Hot-Unplug&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u8%2FSr%2BF8k9qZDO8%2F%2Fg3Eti0OaeJgM5fVPAaw2N7Bf6I%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.phoronix.com%2Fscan.php%3Fpage%3Dnews_item%26px%3DLinux-5.14-AMDGPU-Hot-Unplug&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=u8%2FSr%2BF8k9qZDO8%2F%2Fg3Eti0OaeJgM5fVPAaw2N7Bf6I%3D&reserved=0</a>>><br class=""> <blockquote type="cite" class=""><br class=""> <br class=""> I am not sure about the code part since it all touches KFD driver (KFD<br class=""> team can comment on that) - but I was just wondering if you try 5.14<br class=""> kernel would things just work for you out of the box ?<br class=""> <br class=""> Andrey<br class=""> <br class=""> On 2022-04-05 22:45, Shuotao Xu wrote:<br class=""> <blockquote type="cite" class="">Dear AMD Colleagues,<br class=""> <br class=""> We are from Microsoft Research, and are working on GPU disaggregation<br class=""> technology.<br class=""> <br class=""> We have created a new pull requestAdd PCIe hotplug support for amdgpu by<br class=""> xushuotao · Pull Request #131 · RadeonOpenCompute/ROCK-Kernel-Driver<br class=""> (<a href="http://github.com/" class="">github.com</a>)<br class=""> <<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRadeonOpenCompute%2FROCK-Kernel-Driver%2Fpull%2F131&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tG3fBCB1EYc3dqW9iLsY5d1SYx4bvB7%2BHCuA7HVHzXE%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRadeonOpenCompute%2FROCK-Kernel-Driver%2Fpull%2F131&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tG3fBCB1EYc3dqW9iLsY5d1SYx4bvB7%2BHCuA7HVHzXE%3D&reserved=0</a><br class=""> </blockquote> <<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRadeonOpenCompute%2FROCK-Kernel-Driver%2Fpull%2F131&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tG3fBCB1EYc3dqW9iLsY5d1SYx4bvB7%2BHCuA7HVHzXE%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRadeonOpenCompute%2FROCK-Kernel-Driver%2Fpull%2F131&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tG3fBCB1EYc3dqW9iLsY5d1SYx4bvB7%2BHCuA7HVHzXE%3D&reserved=0</a><span class="Apple-converted-space"> </span><br class=""> </blockquote> <<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRadeonOpenCompute%2FROCK-Kernel-Driver%2Fpull%2F131&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tG3fBCB1EYc3dqW9iLsY5d1SYx4bvB7%2BHCuA7HVHzXE%3D&reserved=0" class="">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRadeonOpenCompute%2FROCK-Kernel-Driver%2Fpull%2F131&data=05%7C01%7Cshuotaoxu%40microsoft.com%7Cc8bd85217a194c7d177808da17e3a825%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637848563851011449%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tG3fBCB1EYc3dqW9iLsY5d1SYx4bvB7%2BHCuA7HVHzXE%3D&reserved=0</a>>>>in<br class=""> <blockquote type="cite" class=""> <blockquote type="cite" class="">ROCK-Kernel-Driver, which will enable PCIe hot-plug support for amdgpu.<br class=""> <br class=""> We believe the support of hot-plug of GPU devices can open doors for<br class=""> many advanced applications in data center in the next few years, and we<br class=""> would like to have some reviewers on this PR so we can continue further<br class=""> technical discussions around this feature.<br class=""> <br class=""> Would you please help review this PR?<br class=""> <br class=""> Thank you very much!<br class=""> <br class=""> Best regards,<br class=""> <br class=""> Shuotao Xu</blockquote> </blockquote> </blockquote> </div> </blockquote> </div> <br class=""> </div> </body> </html>