<!DOCTYPE html><html><head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body> Am 03.04.24 um 14:57 schrieb Jason Gunthorpe: <blockquote type="cite" cite="mid:20240403125712.GA1744080@nvidia.com"> <pre class="moz-quote-pre" wrap="">On Wed, Apr 03, 2024 at 11:16:36AM +0200, Christian König wrote: </pre> <blockquote type="cite"> <pre class="moz-quote-pre" wrap="">Am 03.04.24 um 00:57 schrieb Dave Airlie: </pre> <blockquote type="cite"> <pre class="moz-quote-pre" wrap="">On Wed, 27 Mar 2024 at 19:52, Thomas Hellström <a class="moz-txt-link-rfc2396E" href="mailto:thomas.hellstrom@linux.intel.com"><thomas.hellstrom@linux.intel.com></a> wrote: </pre> <blockquote type="cite"> <pre class="moz-quote-pre" wrap="">Hi! With our SVM mirror work we'll soon start looking at HMM cross-device support. The identified needs are 1) Instead of migrating foreign device memory to system when the current device is faulting, leave it in place... 1a) for access using internal interconnect, 1b) for access using PCIE p2p (probably mostly as a reference) </pre> </blockquote> </blockquote> <pre class="moz-quote-pre" wrap=""> I still agree with Sima that we won't see P2P based on HMM between devices anytime soon if ever. </pre> </blockquote> <pre class="moz-quote-pre" wrap=""> We've got a team working on the subset of this problem where we can have a GPU driver install DEVICE_PRIVATE pages and the RDMA driver use hmm_range_fault() to take the DEVICE_PRIVATE and return an equivilent P2P page for DMA. We already have a working prototype that is not too bad code wise.</pre> </blockquote> The problem with that isn't the software but the hardware. At least on the AMD GPUs and Intels Xe accelerators we have seen so far page faults are not fast enough to actually work with the semantics the Linux kernel uses for struct pages. That's why for example the SVM implementation really suck with fork(), the transparent huge page deamon and NUMA migrations. Somebody should probably sit down and write a performance measurement tool for page faults so that we can start to compare vendors regarding this. <blockquote type="cite" cite="mid:20240403125712.GA1744080@nvidia.com"> <blockquote type="cite"> <pre class="moz-quote-pre" wrap="">E.g. there is no common representation of DMA addresses with address spaces. In other words you need to know the device which does DMA for an address to make sense. </pre> </blockquote> <pre class="moz-quote-pre" wrap=""> ? Every device device calls hmm_range_fault() on it's own, to populate its own private mirror page table, and gets a P2P page. The device can DMA map that P2P for its own use to get a topologically appropriate DMA address for its own private page table. The struct page for P2P references the pgmap which references the target struct device, the DMA API provides the requesting struct device. The infrastructure for all this is all there already.</pre> </blockquote> The problem is the DMA API currently has no idea of inter device connectors like XGMI. So it can create P2P mappings for PCIe, but anything which isn't part of those interconnects is ignore at the moment as far as I can see. <blockquote type="cite" cite="mid:20240403125712.GA1744080@nvidia.com"> <pre class="moz-quote-pre" wrap="">There is a seperate discussion about optimizing away the P2P pgmap, but for the moment I'm focused on getting things working by relying on it. </pre> <blockquote type="cite"> <pre class="moz-quote-pre" wrap="">Additional to that we don't have a representation for internal connections, e.g. the common kernel has no idea that device A and device B can talk directly to each other, but not with device C. </pre> </blockquote> <pre class="moz-quote-pre" wrap=""> We do have this in the PCI P2P framework, it just isn't very complete, but it does handle the immediate cases I see people building where we have switches and ACS/!ACS paths with different addressing depending on topology.</pre> </blockquote> That's not what I meant. I'm talking about direct interconnects which a parallel to the PCIe bus. As far as I know we haven't even started looking into those. <blockquote type="cite" cite="mid:20240403125712.GA1744080@nvidia.com"> <blockquote type="cite"> <blockquote type="cite"> <blockquote type="cite"> <pre class="moz-quote-pre" wrap="">and we plan to add an infrastructure for this. Probably this can be done initially without too much (or any) changes to the hmm code itself. </pre> </blockquote> </blockquote> </blockquote> <pre class="moz-quote-pre" wrap=""> It is essential any work in this area is not tied to DRM. hmm_range_fault() and DEVICE_PRIVATE are generic kernel concepts we need to make them work better not build weird DRM side channels.</pre> </blockquote> Completely agree. <blockquote type="cite" cite="mid:20240403125712.GA1744080@nvidia.com"> <blockquote type="cite"> <blockquote type="cite"> <blockquote type="cite"> <pre class="moz-quote-pre" wrap="">So the question is basically whether anybody is interested in a drm-wide solution for this and in that case also whether anybody sees the need for cross-driver support? </pre> </blockquote> </blockquote> <pre class="moz-quote-pre" wrap=""> We have use cases for this as well, yes. </pre> </blockquote> <pre class="moz-quote-pre" wrap=""> Unfortunately this is a long journey. The immediate next steps are Alistair's work to untangle the DAX refcounting mess from ZONE_DEVICE pages: <a class="moz-txt-link-freetext" href="https://lore.kernel.org/linux-mm/87ttlhmj9p.fsf@nvdebian.thelocal/">https://lore.kernel.org/linux-mm/87ttlhmj9p.fsf@nvdebian.thelocal/</a> Leon is working on improving the DMA API and RDMA's ODP to be better setup for this: <a class="moz-txt-link-freetext" href="https://lore.kernel.org/linux-rdma/cover.1709635535.git.leon@kernel.org/">https://lore.kernel.org/linux-rdma/cover.1709635535.git.leon@kernel.org/</a> [Which is also the basis for fixing DMABUF's abuse of the DMA API] Then it is pretty simple to teach hmm_range_fault() to convert a DEVICE_PRIVATE page into a P2P page using a new pgmap op and from there the rest already basically exists.</pre> </blockquote> Nice, that's at least one step further than I expected. <blockquote type="cite" cite="mid:20240403125712.GA1744080@nvidia.com"> <pre class="moz-quote-pre" wrap="">Folks doing non-PCIe topologies will need to teach the P2P layer how address translation works on those buses.</pre> </blockquote> Where to start with that? Christian. <blockquote type="cite" cite="mid:20240403125712.GA1744080@nvidia.com"> <pre class="moz-quote-pre" wrap=""> Jason </pre> </blockquote> </body> </html>