[PATCH 11/13] drm/amdgpu: add DRM_AMDGPU_ATC config option

[Felix Kuehling]

On 2018-01-30 08:59 AM, Christian König wrote:
> Am 29.01.2018 um 23:08 schrieb Felix Kuehling:
>> On 2018-01-26 03:13 PM, Christian König wrote:
>>> [SNIP]
>>> +#ifdef CONFIG_DRM_AMDGPU_ATC
>>> +    r = amd_iommu_init_device(adev->pdev, 0x10000);
>> KFD queries how many PASIDs the IOMMU can support with
>> amd_iommu_device_info. KFD only assigns PASIDs within that range. It can
>> be much smaller than the 16-bits supported by the GPU.
>>
>> For a VM that uses ATC, you need to make sure it gets a PASID in the
>> range supported by the IOMMU. The PASID manager already supports that
>> and keeps smaller PASIDs for users that really need them.
>
> Yeah, seen that and I'm not really keen about it.
>
> Especially since we need multiple types of PASIDs here:
> 1. For GPUVM debugging and HMM faults, where we can use the full 16bit
> range without worrying about what IOMMU can do.
> 2. For ATC use case where we need to keep the IOMMU in the picture.
>
> Are there any hardware limitations which blocks us from using a per
> device PASID? That would simplify the whole handling quite a bit.

Conceptually PASIDs are device-specific process IDs. KFD uses the same
PASID on all devices, but that's not necessary.

Currently you allocate PASIDs per VM or per open device file. So you'll
end up with different PASIDs on each device. I think you can even end up
with multiple PASIDs in each process even on the same device, if you
create multiple VMs. That doesn't seem to be a problem for the IOMMU driver.

>
> Additional to that we don't really want this direct relationship
> between amdgpu/amdkfd and the amd_iommu_v2 driver.

Not sure what you mean. Right now amdgpu and amdkfd don't coordinate
their PASIDs (other than using a common allocator to avoid using the
same PASID for different things).

>
> So what do you think about moving the PASID handling into the IOMMU
> driver? And abstracting which driver is in use through the iommu_ops?

What if both drivers want to use the IOMMU?

Regards,
  Felix

>
> Regards,
> Christian.
>
>>
>> Regards,
>>    Felix