[PATCH 2/2] drm/amdgpu: Reduce SG bo memory usage for mGPUs

[Felix Kuehling]

On 2021-12-07 9:59 a.m., Philip Yang wrote:
> For userptr bo, if adev is not in IOMMU isolation mode, multiple GPUs
> use same system memory page dma mapping address, they can share the
> original mem->bo in attachment to reduce dma address array memory usage.
>
> Signed-off-by: Philip Yang <Philip.Yang at amd.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 12 ++++++++----
>   1 file changed, 8 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> index b8490789eef4..03807ed1f6e2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> @@ -708,10 +708,14 @@ static int kfd_mem_attach(struct amdgpu_device *adev, struct kgd_mem *mem,
>   		pr_debug("\t add VA 0x%llx - 0x%llx to vm %p\n", va,
>   			 va + bo_size, vm);
>   
> -		if (adev == bo_adev || (mem->domain == AMDGPU_GEM_DOMAIN_VRAM &&
> -					amdgpu_xgmi_same_hive(adev, bo_adev))) {
> -			/* Mappings on the local GPU and VRAM mappings in the
> -			 * local hive share the original BO
> +		if (adev == bo_adev ||
> +		    (amdgpu_ttm_tt_get_usermm(mem->bo->tbo.ttm) &&
> +		     !adev->iommu_isolation) ||
> +		    (mem->domain == AMDGPU_GEM_DOMAIN_VRAM &&
> +		     amdgpu_xgmi_same_hive(adev, bo_adev))) {

For the sake of readability I'd ignore the 80-column rule here (which 
isn't a hard rule any more). It could apply in the same way to GTT. This 
would be clearer:

+		if (adev == bo_adev ||
+		    (amdgpu_ttm_tt_get_usermm(mem->bo->tbo.ttm) && adev->ram_is_direct_mapped) ||
+		    (mem->domain == AMDGPU_GEM_DOMAIN_VRAM && amdgpu_xgmi_same_hive(adev, bo_adev))) {

Regards,
   Felix


> +			/* Mappings on the local GPU, or VRAM mappings in the
> +			 * local hive, or userptr mapping IOMMU is not isolation
> +			 * mode share the original BO
>   			 */
>   			attachment[i]->type = KFD_MEM_ATT_SHARED;
>   			bo[i] = mem->bo;