amd iommu configuration

[Felix Kuehling]

On 2022-09-06 10:52, Robin Murphy wrote:
> On 2022-09-05 15:13, Vasant Hegde wrote:
>> Steven,
>>
>> [+Felix, amd-fgx list]
>>
>>
>> On 9/3/2022 4:29 AM, Steven J Abner wrote:
>>> Hi
>>> I was referred to you from linux-kernel at vger.kernel.org about the 
>>> following issue.
>>> Here is as was written:
>>> On 9/1/22 11:36, Steven J Abner wrote:
>>> Hi
>>> Building a kernel tailored for AMD 2400g on ASRock B450 using 
>>> 5.18.12 as base.
>>> I stumbled across an odd situation and which lacked Kconfig info and 
>>> lead to
>>> oddity.
>>> /drivers/iommu/amd/Kconfig states 'config AMD_IOMMU_V2' is 
>>> 'tristate' but unlike
>>> many
>>> other tristate configures doesn't mention that module name is 
>>> 'iommu_v2.ko' and
>>> loading should be done by adding to modules-load.d.
>>>
>>> The oddity is that by loading as module is as follows (differences):
>>>
>>> builtin iommu_v2 version dmesg:
>>> amdgpu: HMM registered 2048MB device memory
>>> amdgpu: Topology: Add APU node [0x0:0x0]
>>> amdgpu: Topology: Add APU node [0x15dd:0x1002]
>>> AMD-Vi: AMD IOMMUv2 loaded and initialized
>>> kfd kfd: amdgpu: added device 1002:15dd
>>> kfd kfd: amdgpu: Allocated 3969056 bytes on gart
>>> memmap_init_zone_device initialised 524288 pages in 0ms
>>
>> IOMMU V2 modules provides IOMMU feature like attaching device to
>> process. I think amdgpu uses those features if available.
>> So in this case amdgpu is using those IOMMU features.
>>
>>>
>>> module not loaded due to missing iommu.conf dmesg:
>>> amdgpu: CRAT table disabled by module option
>>> amdgpu: Topology: Add CPU node
>>> amdgpu: Virtual CRAT table created for CPU
>>> kfd kfd: amdgpu: GC IP 090100 not supported in kfd
>>>
>>> module load through iommu.conf dmesg:
>>> amdgpu: CRAT table disabled by module option
>>> amdgpu: Topology: Add CPU node
>>> amdgpu: Virtual CRAT table created for CPU
>>> AMD-Vi: AMD IOMMUv2 loaded and initialized
>>> kfd kfd: amdgpu: GC IP 090100 not supported in kfd
>>>
>>> Note, only difference on witk/without iommu.conf is:
>>> AMD-Vi: AMD IOMMUv2 loaded and initialized
>>
>> I think in this case iommu_v2.ko module got loaded after GPU
>> initialized. Hence amdgpu is not using iommu v2 features.
>>
>>
>>>
>>> So does this mean missing features by not having builtin?
>>> If not, should Kconfig have hint about module and loading?
>>
>> @Felix,
>>    I see that drivers/gpu/drm/amd/amdkfd/Kconfig contains below line
>>      imply AMD_IOMMU_V2 if X86_64
>>
>>
>>    Should we change `s/imply/select` ?
>
> "select" might help when KFD is built-in, but it probably still wants 
> a MODULE_SOFTDEP() to enforce load order when they're both modular.

We don't want to make it a hard dependency because only a small subset 
of all AMD GPUs use iommuv2, and we have fallbacks in place for even 
those to work without it.

Regards,
   Felix


>
> Robin.