[PATCH 2/3] drm/amdgpu: Use preemptible domain for KFD

[Felix Kuehling]

Hi Xinhui,

This should be another way to fix the userptr swapout problem you found. 
This is meant to address a long-stadinp problem with userptr BOs getting 
evicted (not swapped), which broke DMA mappings. This patch series is 
meant to completely avoid evictions of userptrs and dma mappings for 
preemptible KFD contexts. That should also avoid the swapout issue.

If there is an underlying problem with the way TTM handles swapout of 
userptrs, that would still affect userptrs in the graphics driver, though.

Regards,
   Felix

On 2021-05-19 1:45 a.m., Felix Kuehling wrote:
> KFD userptr BOs and SG BOs used for DMA mappings can be preempted with
> CWSR. Therefore we can place them in the preemptible domain and avoid
> unwanted evictions.
>
> Signed-off-by: Felix Kuehling <Felix.Kuehling at amd.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> index 928e8d57cd08..2856ca1032db 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
> @@ -511,7 +511,7 @@ kfd_mem_dmamap_userptr(struct kgd_mem *mem,
>   	drm_prime_sg_to_dma_addr_array(ttm->sg, ttm->dma_address,
>   				       ttm->num_pages);
>   
> -	amdgpu_bo_placement_from_domain(bo, AMDGPU_GEM_DOMAIN_GTT);
> +	amdgpu_bo_placement_from_domain(bo, AMDGPU_GEM_DOMAIN_PREEMPT);
>   	ret = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx);
>   	if (ret)
>   		goto unmap_sg;
> @@ -535,7 +535,7 @@ kfd_mem_dmamap_dmabuf(struct kfd_mem_attachment *attachment)
>   	struct ttm_operation_ctx ctx = {.interruptible = true};
>   	struct amdgpu_bo *bo = attachment->bo_va->base.bo;
>   
> -	amdgpu_bo_placement_from_domain(bo, AMDGPU_GEM_DOMAIN_GTT);
> +	amdgpu_bo_placement_from_domain(bo, AMDGPU_GEM_DOMAIN_PREEMPT);
>   	return ttm_bo_validate(&bo->tbo, &bo->placement, &ctx);
>   }
>   
> @@ -1408,7 +1408,7 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
>   		domain = alloc_domain = AMDGPU_GEM_DOMAIN_GTT;
>   		alloc_flags = 0;
>   	} else if (flags & KFD_IOC_ALLOC_MEM_FLAGS_USERPTR) {
> -		domain = AMDGPU_GEM_DOMAIN_GTT;
> +		domain = AMDGPU_GEM_DOMAIN_PREEMPT;
>   		alloc_domain = AMDGPU_GEM_DOMAIN_CPU;
>   		alloc_flags = 0;
>   		if (!offset || !*offset)