[PATCH] drm/amdgpu: Add preferred_domain check when determine XGMI state

[Kuehling, Felix]

On 2019-03-26 2:54 p.m., Liu, Shaoyun wrote:
> Avoid unnecessary XGMI hight pstate trigger when mapping none-vram memory for peer device
>
> Change-Id: I1881deff3da19f1f4b58d5765db03a590092a5b2
> Signed-off-by: shaoyunl <shaoyun.liu at amd.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 9 +++++++++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c  | 3 ++-
>   2 files changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> index a82c3b1..3c7ee71 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c
> @@ -664,8 +664,10 @@ int amdgpu_gem_op_ioctl(struct drm_device *dev, void *data,
>   			struct drm_file *filp)
>   {
>   	struct amdgpu_device *adev = dev->dev_private;
> +	struct amdgpu_fpriv *fpriv = filp->driver_priv;
>   	struct drm_amdgpu_gem_op *args = data;
>   	struct drm_gem_object *gobj;
> +	struct amdgpu_bo_va *bo_va;
>   	struct amdgpu_bo *robj;
>   	int r;
>   
> @@ -704,6 +706,13 @@ int amdgpu_gem_op_ioctl(struct drm_device *dev, void *data,
>   			amdgpu_bo_unreserve(robj);
>   			break;
>   		}
> +		bo_va = amdgpu_vm_bo_find(&fpriv->vm, robj);
> +		if (bo_va && bo_va->is_xgmi) {
> +			r = -EINVAL;
> +			amdgpu_bo_unreserve(robj);
> +			break;
> +		}
> +

Hmm, from the other discussion, GEM doesn't really support P2P of VRAM 
BOs between GPUs right now. The only way this function can affect a BO 
that's P2P shared is, if the BO is allocated with GEM and then imported 
into KFD. In that case you'll need to take into account mappings of the 
imported BO in all the KFD VMs, not the VMs in the fpriv->vm.

In other words, you need to find all bo_vas of the BO in all VMs and for 
each one check, whether it has is_xgmi set.

Regards,
   Felix


>   		robj->preferred_domains = args->value & (AMDGPU_GEM_DOMAIN_VRAM |
>   							AMDGPU_GEM_DOMAIN_GTT |
>   							AMDGPU_GEM_DOMAIN_CPU);
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> index 76eee7e..f08dda2 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> @@ -2048,7 +2048,8 @@ struct amdgpu_bo_va *amdgpu_vm_bo_add(struct amdgpu_device *adev,
>   	INIT_LIST_HEAD(&bo_va->valids);
>   	INIT_LIST_HEAD(&bo_va->invalids);
>   
> -	if (bo && amdgpu_xgmi_same_hive(adev, amdgpu_ttm_adev(bo->tbo.bdev))) {
> +	if (bo && amdgpu_xgmi_same_hive(adev, amdgpu_ttm_adev(bo->tbo.bdev)) &&
> +	    (bo->preferred_domains == AMDGPU_GEM_DOMAIN_VRAM)) {
>   		bo_va->is_xgmi = true;
>   		mutex_lock(&adev->vm_manager.lock_pstate);
>   		/* Power up XGMI if it can be potentially used */