[PATCH 2/2] drm/amdgpu:impl virt_gart_flush_tlbs

[Christian König]

Am 13.12.2017 um 04:42 schrieb Monk Liu:
> a new gart flush tlb function implemented for SRIOV,
> and invoke it during RUNTIME for gart flush TLBs
>
> this could avoid the issue that gart flush (via CPU MMIO)
> being interrupted by word switch which lead to DMAR error
> on Host/IOMMU side, with this function the gart flush
> tlbs always run on KIQ with single PM4 package so it won't
> get interrupted before the flushing finished.
>
> Change-Id: I0849658d7945c3874b3cc0d9369a50e1aedb8312
> Signed-off-by: Monk Liu <Monk.Liu at amd.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c | 27 +++++++++++++++++++++++++++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h |  1 +
>   drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c    |  3 +++
>   3 files changed, 31 insertions(+)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c
> index e7dfb7b..7a6ef64 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c
> @@ -172,6 +172,33 @@ void amdgpu_virt_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v)
>   		DRM_ERROR("wait for kiq fence error: %ld\n", r);
>   }
>   
> +int amdgpu_virt_gart_flush_tlbs(struct amdgpu_device *adev)
> +{
> +	struct amdgpu_kiq *kiq = &adev->gfx.kiq;
> +	struct amdgpu_ring *ring = &kiq->ring;
> +	unsigned long flags;
> +	signed long r;
> +	uint32_t seq;
> +
> +	if(!ring->funcs->emit_invalidate_tlbs)
> +		return -ENOENT;
> +
> +	spin_lock_irqsave(&kiq->ring_lock, flags);
> +	amdgpu_ring_alloc(ring, 16);
> +	amdgpu_ring_emit_invalidate_tlbs(ring);
> +	amdgpu_fence_emit_polling(ring, &seq);
> +	amdgpu_ring_commit(ring);
> +	spin_unlock_irqrestore(&kiq->ring_lock, flags);
> +
> +	r = amdgpu_fence_wait_polling(ring, seq, MAX_KIQ_REG_WAIT);
> +	if (r < 1) {
> +		DRM_ERROR("wait for kiq invalidate tlbs error: %ld\n", r);
> +		return -ETIME;
> +	}
> +
> +	return 0;
> +}
> +
>   /**
>    * amdgpu_virt_request_full_gpu() - request full gpu access
>    * @amdgpu:	amdgpu device.
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> index 6a83425..935fed3 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h
> @@ -297,5 +297,6 @@ int amdgpu_virt_fw_reserve_get_checksum(void *obj, unsigned long obj_size,
>   					unsigned int key,
>   					unsigned int chksum);
>   void amdgpu_virt_init_data_exchange(struct amdgpu_device *adev);
> +int amdgpu_virt_gart_flush_tlbs(struct amdgpu_device *adev);
>   
>   #endif
> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> index 1b5dfcc..a195039 100644
> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
> @@ -332,6 +332,9 @@ static void gmc_v9_0_gart_flush_gpu_tlb(struct amdgpu_device *adev,
>   	/* flush hdp cache */
>   	adev->nbio_funcs->hdp_flush(adev);
>   
> +	if (amdgpu_sriov_runtime(adev) && !amdgpu_virt_gart_flush_tlbs(adev))
> +		return;
> +

Better open code that like this:

if (amdgpu_sriov_runtime(adev)) {
     /* Try using the KIQ */
     r = amdgpu_virt_gart_flush_tlbs(adev));
     if (!r)
         return;
}

Apart from that it looks good to me.

Christian.

>   	spin_lock(&adev->mc.invalidate_lock);
>   
>   	for (i = 0; i < AMDGPU_MAX_VMHUBS; ++i) {