[PATCH 1/3] drm/amdkfd: Define config HSA_AMD_P2P to support peer-to-peer

[Errabolu, Ramesh]

[AMD Official Use Only - General]

Update the both the description and commit message per review comments.
Will post new patch shortly.

Regards,
Ramesh

-----Original Message-----
From: Kuehling, Felix <Felix.Kuehling at amd.com> 
Sent: Wednesday, June 1, 2022 9:15 PM
To: Errabolu, Ramesh <Ramesh.Errabolu at amd.com>; amd-gfx at lists.freedesktop.org
Subject: Re: [PATCH 1/3] drm/amdkfd: Define config HSA_AMD_P2P to support peer-to-peer

Am 2022-05-31 um 13:02 schrieb Ramesh Errabolu:
> Extend current kernel config requirements of amdgpu by adding config 
> HSA_AMD_P2P. Enabling HSA_AMD_P2P is REQUIRED to support peer-to-peer 
> communication, in both data and control planes, among AMD GPU devices 
> that are connected via PCIe and have large BAR vBIOS
>
> Signed-off-by: Ramesh Errabolu <Ramesh.Errabolu at amd.com>
> ---
>   drivers/gpu/drm/amd/amdkfd/Kconfig | 8 ++++++++
>   1 file changed, 8 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig 
> b/drivers/gpu/drm/amd/amdkfd/Kconfig
> index 8cc0a76ddf9f..26614f5f20ea 100644
> --- a/drivers/gpu/drm/amd/amdkfd/Kconfig
> +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
> @@ -25,3 +25,11 @@ config HSA_AMD_SVM
>   	  preemptions and one based on page faults. To enable page fault
>   	  based memory management on most GFXv9 GPUs, set the module
>   	  parameter amdgpu.noretry=0.
> +
> +config HSA_AMD_P2P
> +	bool "HSA kernel driver support for peer-to-peer for AMD GPU devices"
> +	depends on HSA_AMD && PCI_P2PDMA && DMABUF_MOVE_NOTIFY
> +	help
> +	  Enable this if you want to access AMD GPU peer devices, in both data
> +	  and control planes, that are connected via PCIe and have large BAR 
> +vBIOS

I have not seen the terms "data plane" and "control plane" used in the context of GPUs. As far as I can tell, this terminology is more common in the context network routing. I think it could cause confusion to introduce these terms without an explanation to users.

The sentence "... if you want to access AMD GPU peer devices ..." seems to address someone writing an application. This help message is meant for users and admins building a kernel, who may want to run compute applications, not for compute application developers.

I would also not mention large-BAR VBIOSes because the BAR can often be resized even with a small-BAR VBIOS.

Therefore I would recommend an alternative text here that avoids uncommon terminology and addresses the concerns of users rather than application developers:

    Enable peer-to-peer (P2P) communication between AMD GPUs over the
    PCIe bus. This can improve performance of multi-GPU compute
    applications and libraries by enabling GPUs to access data directly
    in peer GPUs' memory without intermediate copies in system memory.

    This P2P feature is only enabled on compatible chipsets, and between
    GPUs with large memory BARs that expose the entire VRAM in PCI bus
    address space within the physical address limits of the GPUs.

Regards,
   Felix


> +