[PATCH 1/3] drm/amdgpu: Add userptr bo support for mGPUs when iommu is on

[Felix Kuehling]

On 2023-04-05 11:31, Shane Xiao wrote:
> For userptr bo with iommu on, multiple GPUs use same system
> memory dma mapping address when both adev and bo_adev are in
> identity mode or in the same iommu group.
>
> If RAM direct map to one GPU, other GPUs can share the original
> BO in order to reduce dma address array usage when RAM can
> direct map to these GPUs. However, we should explicit check
> whether RAM can direct map to all these GPUs.
>
> This patch fixes a potential issue that where RAM is
> direct mapped on some but not all GPUs.
>
> v2:
>    1. Update comment
>    2. Add helper function reuse_dmamap
>
> Signed-off-by: Shane Xiao <shane.xiao at amd.com>

Reviewed-by: Felix Kuehling <Felix.Kuehling at amd.com>


> ---
>   .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  | 27 ++++++++++++++++---
>   1 file changed, 23 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> index e7403f8e4eba..2fbb9db64ad5 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> @@ -81,6 +81,25 @@ static bool kfd_mem_is_attached(struct amdgpu_vm *avm,
>   	return false;
>   }
>   
> +/**
> + * reuse_dmamap() - Check whether adev can share the original
> + * userptr BO
> + *
> + * If both adev and bo_adev are in direct mapping or
> + * in the same iommu group, they can share the original BO.
> + *
> + * @adev: Device to which can or cannot share the original BO
> + * @bo_adev: Device to which allocated BO belongs to
> + *
> + * Return: returns true if adev can share original userptr BO,
> + * false otherwise.
> + */
> +static bool reuse_dmamap(struct amdgpu_device *adev, struct amdgpu_device *bo_adev)
> +{
> +	return (adev->ram_is_direct_mapped && bo_adev->ram_is_direct_mapped) ||
> +			(adev->dev->iommu_group == bo_adev->dev->iommu_group);
> +}
> +
>   /* Set memory usage limits. Current, limits are
>    *  System (TTM + userptr) memory - 15/16th System RAM
>    *  TTM memory - 3/8th System RAM
> @@ -804,11 +823,11 @@ static int kfd_mem_attach(struct amdgpu_device *adev, struct kgd_mem *mem,
>   			 va + bo_size, vm);
>   
>   		if ((adev == bo_adev && !(mem->alloc_flags & KFD_IOC_ALLOC_MEM_FLAGS_MMIO_REMAP)) ||
> -		    (amdgpu_ttm_tt_get_usermm(mem->bo->tbo.ttm) && adev->ram_is_direct_mapped) ||
> -		    same_hive) {
> +		    (amdgpu_ttm_tt_get_usermm(mem->bo->tbo.ttm) && reuse_dmamap(adev, bo_adev)) ||
> +			same_hive) {
>   			/* Mappings on the local GPU, or VRAM mappings in the
> -			 * local hive, or userptr mapping IOMMU direct map mode
> -			 * share the original BO
> +			 * local hive, or userptr mapping can reuse dma map
> +			 * address space share the original BO
>   			 */
>   			attachment[i]->type = KFD_MEM_ATT_SHARED;
>   			bo[i] = mem->bo;