amd iommu configuration

[Robin Murphy]

On 2022-09-05 15:13, Vasant Hegde wrote:
> Steven,
> 
> [+Felix, amd-fgx list]
> 
> 
> On 9/3/2022 4:29 AM, Steven J Abner wrote:
>> Hi
>> I was referred to you from linux-kernel at vger.kernel.org about the following issue.
>> Here is as was written:
>> On 9/1/22 11:36, Steven J Abner wrote:
>> Hi
>> Building a kernel tailored for AMD 2400g on ASRock B450 using 5.18.12 as base.
>> I stumbled across an odd situation and which lacked Kconfig info and lead to
>> oddity.
>> /drivers/iommu/amd/Kconfig states 'config AMD_IOMMU_V2' is 'tristate' but unlike
>> many
>> other tristate configures doesn't mention that module name is 'iommu_v2.ko' and
>> loading should be done by adding to modules-load.d.
>>
>> The oddity is that by loading as module is as follows (differences):
>>
>> builtin iommu_v2 version dmesg:
>> amdgpu: HMM registered 2048MB device memory
>> amdgpu: Topology: Add APU node [0x0:0x0]
>> amdgpu: Topology: Add APU node [0x15dd:0x1002]
>> AMD-Vi: AMD IOMMUv2 loaded and initialized
>> kfd kfd: amdgpu: added device 1002:15dd
>> kfd kfd: amdgpu: Allocated 3969056 bytes on gart
>> memmap_init_zone_device initialised 524288 pages in 0ms
> 
> IOMMU V2 modules provides IOMMU feature like attaching device to
> process. I think amdgpu uses those features if available.
> So in this case amdgpu is using those IOMMU features.
> 
>>
>> module not loaded due to missing iommu.conf dmesg:
>> amdgpu: CRAT table disabled by module option
>> amdgpu: Topology: Add CPU node
>> amdgpu: Virtual CRAT table created for CPU
>> kfd kfd: amdgpu: GC IP 090100 not supported in kfd
>>
>> module load through iommu.conf dmesg:
>> amdgpu: CRAT table disabled by module option
>> amdgpu: Topology: Add CPU node
>> amdgpu: Virtual CRAT table created for CPU
>> AMD-Vi: AMD IOMMUv2 loaded and initialized
>> kfd kfd: amdgpu: GC IP 090100 not supported in kfd
>>
>> Note, only difference on witk/without iommu.conf is:
>> AMD-Vi: AMD IOMMUv2 loaded and initialized
> 
> I think in this case iommu_v2.ko module got loaded after GPU
> initialized. Hence amdgpu is not using iommu v2 features.
> 
> 
>>
>> So does this mean missing features by not having builtin?
>> If not, should Kconfig have hint about module and loading?
> 
> @Felix,
>    I see that drivers/gpu/drm/amd/amdkfd/Kconfig contains below line
>      imply AMD_IOMMU_V2 if X86_64
> 
> 
>    Should we change `s/imply/select` ?

"select" might help when KFD is built-in, but it probably still wants a 
MODULE_SOFTDEP() to enforce load order when they're both modular.

Robin.