[PATCH 1/3] drm/amdgpu: Add userptr bo support for mGPUs when iommu is on

[Xiao, Shane]

[AMD Official Use Only - General]



> -----Original Message-----
> From: Koenig, Christian <Christian.Koenig at amd.com>
> Sent: Tuesday, April 4, 2023 6:27 PM
> To: Xiao, Shane <shane.xiao at amd.com>; amd-gfx at lists.freedesktop.org;
> Kuehling, Felix <Felix.Kuehling at amd.com>
> Cc: Liu, Aaron <Aaron.Liu at amd.com>; Guo, Shikai <Shikai.Guo at amd.com>
> Subject: Re: [PATCH 1/3] drm/amdgpu: Add userptr bo support for mGPUs
> when iommu is on
> 
> Am 04.04.23 um 11:56 schrieb Shane Xiao:
> > For userptr bo with iommu on, multiple GPUs use same system memory dma
> > mapping address when both bo_adev and adev in identity mode or in the
> > same iommu group.

Hi Christian,

> 
> WTF? Userptr BOs are not allowed to be exported/imported between different
> GPUs.
> 
> So how can the same userptr BO be used on different GPUs?

If GPUs are all in iommu identity mode which means dma address are the same as physical address,  all of the GPUs can see the system memory directly.

In such case, should we export/import the BO,  then create a new SG BO for another GPU? 


Best Regards,
Shane

> 
> Regards,
> Christian.
> 
> >
> > Signed-off-by: Shane Xiao <shane.xiao at amd.com>
> > ---
> >   drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 8 ++++----
> >   1 file changed, 4 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> > index e7403f8e4eba..33cda358cb9e 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> > @@ -804,11 +804,11 @@ static int kfd_mem_attach(struct amdgpu_device
> *adev, struct kgd_mem *mem,
> >   			 va + bo_size, vm);
> >
> >   		if ((adev == bo_adev && !(mem->alloc_flags &
> KFD_IOC_ALLOC_MEM_FLAGS_MMIO_REMAP)) ||
> > -		    (amdgpu_ttm_tt_get_usermm(mem->bo->tbo.ttm) &&
> adev->ram_is_direct_mapped) ||
> > -		    same_hive) {
> > +		    (amdgpu_ttm_tt_get_usermm(mem->bo->tbo.ttm) &&
> ((adev->ram_is_direct_mapped && bo_adev->ram_is_direct_mapped) ||
> > +			adev->dev->iommu_group == bo_adev->dev-
> >iommu_group)) ||
> > +same_hive){
> >   			/* Mappings on the local GPU, or VRAM mappings in
> the
> > -			 * local hive, or userptr mapping IOMMU direct map
> mode
> > -			 * share the original BO
> > +			 * local hive, or userptr mapping in the same dma
> > +			 * address space share the original BO
> >   			 */
> >   			attachment[i]->type = KFD_MEM_ATT_SHARED;
> >   			bo[i] = mem->bo;