[PATCH 1/3] drm/amdkfd: Define config HSA_AMD_P2P to support peer-to-peer

[Felix Kuehling]

Am 2022-05-31 um 13:02 schrieb Ramesh Errabolu:
> Extend current kernel config requirements of amdgpu by adding
> config HSA_AMD_P2P. Enabling HSA_AMD_P2P is REQUIRED to support
> peer-to-peer communication, in both data and control planes, among
> AMD GPU devices that are connected via PCIe and have large BAR vBIOS
>
> Signed-off-by: Ramesh Errabolu <Ramesh.Errabolu at amd.com>
> ---
>   drivers/gpu/drm/amd/amdkfd/Kconfig | 8 ++++++++
>   1 file changed, 8 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
> index 8cc0a76ddf9f..26614f5f20ea 100644
> --- a/drivers/gpu/drm/amd/amdkfd/Kconfig
> +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
> @@ -25,3 +25,11 @@ config HSA_AMD_SVM
>   	  preemptions and one based on page faults. To enable page fault
>   	  based memory management on most GFXv9 GPUs, set the module
>   	  parameter amdgpu.noretry=0.
> +
> +config HSA_AMD_P2P
> +	bool "HSA kernel driver support for peer-to-peer for AMD GPU devices"
> +	depends on HSA_AMD && PCI_P2PDMA && DMABUF_MOVE_NOTIFY
> +	help
> +	  Enable this if you want to access AMD GPU peer devices, in both data
> +	  and control planes, that are connected via PCIe and have large BAR vBIOS

I have not seen the terms "data plane" and "control plane" used in the 
context of GPUs. As far as I can tell, this terminology is more common 
in the context network routing. I think it could cause confusion to 
introduce these terms without an explanation to users.

The sentence "... if you want to access AMD GPU peer devices ..." seems 
to address someone writing an application. This help message is meant 
for users and admins building a kernel, who may want to run compute 
applications, not for compute application developers.

I would also not mention large-BAR VBIOSes because the BAR can often be 
resized even with a small-BAR VBIOS.

Therefore I would recommend an alternative text here that avoids 
uncommon terminology and addresses the concerns of users rather than 
application developers:

    Enable peer-to-peer (P2P) communication between AMD GPUs over the
    PCIe bus. This can improve performance of multi-GPU compute
    applications and libraries by enabling GPUs to access data directly
    in peer GPUs' memory without intermediate copies in system memory.

    This P2P feature is only enabled on compatible chipsets, and between
    GPUs with large memory BARs that expose the entire VRAM in PCI bus
    address space within the physical address limits of the GPUs.

Regards,
   Felix


> +