Cross-device and cross-driver HMM support

[Christian König]

Am 03.04.24 um 14:57 schrieb Jason Gunthorpe:
> On Wed, Apr 03, 2024 at 11:16:36AM +0200, Christian König wrote:
>> Am 03.04.24 um 00:57 schrieb Dave Airlie:
>>> On Wed, 27 Mar 2024 at 19:52, Thomas Hellström
>>> <thomas.hellstrom at linux.intel.com>  wrote:
>>>> Hi!
>>>>
>>>> With our SVM mirror work we'll soon start looking at HMM cross-device
>>>> support. The identified needs are
>>>>
>>>> 1) Instead of migrating foreign device memory to system when the
>>>> current device is faulting, leave it in place...
>>>> 1a) for access using internal interconnect,
>>>> 1b) for access using PCIE p2p (probably mostly as a reference)
>> I still agree with Sima that we won't see P2P based on HMM between devices
>> anytime soon if ever.
> We've got a team working on the subset of this problem where we can
> have a GPU driver install DEVICE_PRIVATE pages and the RDMA driver use
> hmm_range_fault() to take the DEVICE_PRIVATE and return an equivilent
> P2P page for DMA.
>
> We already have a working prototype that is not too bad code wise.

The problem with that isn't the software but the hardware.

At least on the AMD GPUs and Intels Xe accelerators we have seen so far 
page faults are not fast enough to actually work with the semantics the 
Linux kernel uses for struct pages.

That's why for example the SVM implementation really suck with fork(), 
the transparent huge page deamon and NUMA migrations.

Somebody should probably sit down and write a performance measurement 
tool for page faults so that we can start to compare vendors regarding this.

>> E.g. there is no common representation of DMA addresses with address spaces.
>> In other words you need to know the device which does DMA for an address to
>> make sense.
> ? Every device device calls hmm_range_fault() on it's own, to populate
> its own private mirror page table, and gets a P2P page. The device can
> DMA map that P2P for its own use to get a topologically appropriate
> DMA address for its own private page table. The struct page for P2P
> references the pgmap which references the target struct device, the
> DMA API provides the requesting struct device. The infrastructure for
> all this is all there already.

The problem is the DMA API currently has no idea of inter device 
connectors like XGMI.

So it can create P2P mappings for PCIe, but anything which isn't part of 
those interconnects is ignore at the moment as far as I can see.

> There is a seperate discussion about optimizing away the P2P pgmap,
> but for the moment I'm focused on getting things working by relying on
> it.
>
>> Additional to that we don't have a representation for internal connections,
>> e.g. the common kernel has no idea that device A and device B can talk
>> directly to each other, but not with device C.
> We do have this in the PCI P2P framework, it just isn't very complete,
> but it does handle the immediate cases I see people building where we
> have switches and ACS/!ACS paths with different addressing depending
> on topology.

That's not what I meant. I'm talking about direct interconnects which a 
parallel to the PCIe bus.

As far as I know we haven't even started looking into those.

>>>> and we plan to add an infrastructure for this. Probably this can be
>>>> done initially without too much (or any) changes to the hmm code
>>>> itself.
> It is essential any work in this area is not tied to DRM.
> hmm_range_fault() and DEVICE_PRIVATE are generic kernel concepts we
> need to make them work better not build weird DRM side channels.

Completely agree.

>>>> So the question is basically whether anybody is interested in a
>>>> drm-wide solution for this and in that case also whether anybody sees
>>>> the need for cross-driver support?
>> We have use cases for this as well, yes.
> Unfortunately this is a long journey. The immediate next steps are
> Alistair's work to untangle the DAX refcounting mess from ZONE_DEVICE
> pages:
>
> https://lore.kernel.org/linux-mm/87ttlhmj9p.fsf@nvdebian.thelocal/
>
> Leon is working on improving the DMA API and RDMA's ODP to
> be better setup for this:
>
> https://lore.kernel.org/linux-rdma/cover.1709635535.git.leon@kernel.org/
>
> [Which is also the basis for fixing DMABUF's abuse of the DMA API]
>
> Then it is pretty simple to teach hmm_range_fault() to convert a
> DEVICE_PRIVATE page into a P2P page using a new pgmap op and from
> there the rest already basically exists.

Nice, that's at least one step further than I expected.

> Folks doing non-PCIe topologies will need to teach the P2P layer how
> address translation works on those buses.

Where to start with that?

Christian.

>
> Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/intel-xe/attachments/20240403/5d8cf2f3/attachment-0001.htm>