[PATCH] drm/amdgpu: ensure no PCIe peer access for CPU XGMI iolinks

[Felix Kuehling]

Am 2022-08-28 um 11:28 schrieb Christian König:
> Am 26.08.22 um 23:49 schrieb Felix Kuehling:
>> On 2022-08-26 11:47, Alex Sierra wrote:
>>> [Why] Devices with CPU XGMI iolink do not support PCIe peer access.
>>>
>>> Signed-off-by: Alex Sierra <alex.sierra at amd.com>
>>> ---
>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 ++-
>>>   1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> index ce7d117efdb5..1ff66718639d 100644
>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>> @@ -5538,7 +5538,8 @@ bool amdgpu_device_is_peer_accessible(struct 
>>> amdgpu_device *adev,
>>>       return pcie_p2p && p2p_access && (adev->gmc.visible_vram_size &&
>>>           adev->gmc.real_vram_size == adev->gmc.visible_vram_size &&
>>>           !(adev->gmc.aper_base & address_mask ||
>>> -          aper_limit & address_mask));
>>> +          aper_limit & address_mask) &&
>>> +        !adev->gmc.xgmi.connected_to_cpu);
>>
>> Maybe it would be more readable if you added this to the 
>> initialization of p2p_access. In that case you can also save yourself 
>> the call to pci_p2pdma_distance_many and all the aperture checks. 
>> Like this:
>>
>>     bool p2p_access = !adev->gmc.xgmi.connected_to_cpu &&
>>         !(pci_p2pdma_distance_many(adev->pdev, &peer_adev->dev, 1, 
>> true) < 0);
>
> Yeah, agree. Static checks should come first.
>
> But another question: When we have an XGMI link to the CPU all local 
> memory is directly accessible, isn't it?
>
> So the PCIe root complex should be able to allow PCIe device to access 
> our local memory, even with snooping.
>
> Or do I miss something here?

This patch is meant for the case that the module parameter 
amdgpu_use_xgmi_p2p=0. With XGMI disabled, I don't think PCIe P2P would 
work on this platform. This patch will ensure that we don't report PCIe 
P2P links in the KFD topology, and don't attempt to create PCIe P2P 
mappings and potentially DMA mappings in the page tables.

Regards,
   Felix


>
> Regards,
> Christian.
>
>>
>> Regards,
>>   Felix
>>
>>
>>>   #else
>>>       return false;
>>>   #endif
>