[PATCH 11/13] drm/amdgpu: add DRM_AMDGPU_ATC config option

[Oded Gabbay]

On Tue, Jan 30, 2018 at 6:53 PM, Felix Kuehling <felix.kuehling at amd.com> wrote:
>
> On 2018-01-30 08:59 AM, Christian König wrote:
>> Am 29.01.2018 um 23:08 schrieb Felix Kuehling:
>>> On 2018-01-26 03:13 PM, Christian König wrote:
>>>> [SNIP]
>>>> +#ifdef CONFIG_DRM_AMDGPU_ATC
>>>> +    r = amd_iommu_init_device(adev->pdev, 0x10000);
>>> KFD queries how many PASIDs the IOMMU can support with
>>> amd_iommu_device_info. KFD only assigns PASIDs within that range. It can
>>> be much smaller than the 16-bits supported by the GPU.
>>>
>>> For a VM that uses ATC, you need to make sure it gets a PASID in the
>>> range supported by the IOMMU. The PASID manager already supports that
>>> and keeps smaller PASIDs for users that really need them.
>>
>> Yeah, seen that and I'm not really keen about it.
>>
>> Especially since we need multiple types of PASIDs here:
>> 1. For GPUVM debugging and HMM faults, where we can use the full 16bit
>> range without worrying about what IOMMU can do.
>> 2. For ATC use case where we need to keep the IOMMU in the picture.
>>
>> Are there any hardware limitations which blocks us from using a per
>> device PASID? That would simplify the whole handling quite a bit.
>
> Conceptually PASIDs are device-specific process IDs. KFD uses the same
> PASID on all devices, but that's not necessary.
>
> Currently you allocate PASIDs per VM or per open device file. So you'll
> end up with different PASIDs on each device. I think you can even end up
> with multiple PASIDs in each process even on the same device, if you
> create multiple VMs. That doesn't seem to be a problem for the IOMMU driver.
>
>>
>> Additional to that we don't really want this direct relationship
>> between amdgpu/amdkfd and the amd_iommu_v2 driver.
>
> Not sure what you mean. Right now amdgpu and amdkfd don't coordinate
> their PASIDs (other than using a common allocator to avoid using the
> same PASID for different things).
>
>>
>> So what do you think about moving the PASID handling into the IOMMU
>> driver? And abstracting which driver is in use through the iommu_ops?
>
> What if both drivers want to use the IOMMU?
>

There was some discussion last year about a generic PASID allocator in
the iommu subsystem:
https://lists.linuxfoundation.org/pipermail/iommu/2017-June/022159.html

Unfortunately, I don't think it got any further then that.

Oded

> Regards,
>   Felix
>
>>
>> Regards,
>> Christian.
>>
>>>
>>> Regards,
>>>    Felix
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx