[PATCH 4/5] drm/amdgpu: Support snooped PTE flag

[Koenig, Christian]

Am 09.08.19 um 16:24 schrieb Zeng, Oak:
>
> Regards,
> Oak
>
> -----Original Message-----
> From: Koenig, Christian <Christian.Koenig at amd.com>
> Sent: Friday, August 9, 2019 8:31 AM
> To: Zeng, Oak <Oak.Zeng at amd.com>; amd-gfx at lists.freedesktop.org
> Cc: Kuehling, Felix <Felix.Kuehling at amd.com>; Keely, Sean <Sean.Keely at amd.com>
> Subject: Re: [PATCH 4/5] drm/amdgpu: Support snooped PTE flag
>
> Am 09.08.19 um 04:15 schrieb Zeng, Oak:
>> Set snooped PTE flag according to mapping flag. Write request to a
>> page with snooped bit set, will send out invalidate probe request to
>> TCC of the remote GPU where the vram page resides.
>>
>> Change-Id: I799f68ec7a5a1abf32075f5ef31051641a0b3736
>> Signed-off-by: Oak Zeng <Oak.Zeng at amd.com>
>> ---
>>    drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 3 +++
>>    1 file changed, 3 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>> b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>> index d709902..8faead3 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>> @@ -599,6 +599,9 @@ static uint64_t gmc_v9_0_get_vm_pte_flags(struct amdgpu_device *adev,
>>    	if (flags & AMDGPU_VM_PAGE_PRT)
>>    		pte_flag |= AMDGPU_PTE_PRT;
>>    
>> +	if (flags & AMDGPU_VM_PAGE_INVALIDATE_PROBE)
>> +		pte_flag |= AMDGPU_PTE_SNOOPED;
>> +
> Still a NAK, we absolutely need a check here that this is only set when the BO is in XGMI.
>
> [Oak]: Per discussion with upper layer stack, remote vram mapping (either over PCIe or XGMI) should always invalidate probe the cache lines of GPU who owns the memory object.

Then we can just drop the AMDGPU_VM_PAGE_INVALIDATE_PROBE flag.

But the problem is something entirely different, the PTE_SNOOPED flag 
has different meaning depending on if the transaction is routed to XGMI 
or PCIe.

For XGMI the flag triggers invalidate probe on the remote GPU, but for 
PCIe it controls CPU cache snooping.

And the CPU cache snooping flags of the devices and host must match or 
otherwise you run into a hell lot of problems on some architectures.

So we can only allow setting the snooped flag here when that PTE is 
really pointing to XGMI.

Regards,
Christian.

>
> Christian.
>
>>    	return pte_flag;
>>    }
>>