amd iommu configuration

[Vasant Hegde]

Steven,

[+Felix, amd-fgx list]


On 9/3/2022 4:29 AM, Steven J Abner wrote:
> Hi
> I was referred to you from linux-kernel at vger.kernel.org about the following issue.
> Here is as was written:
> On 9/1/22 11:36, Steven J Abner wrote:
> Hi
> Building a kernel tailored for AMD 2400g on ASRock B450 using 5.18.12 as base.
> I stumbled across an odd situation and which lacked Kconfig info and lead to
> oddity.
> /drivers/iommu/amd/Kconfig states 'config AMD_IOMMU_V2' is 'tristate' but unlike
> many
> other tristate configures doesn't mention that module name is 'iommu_v2.ko' and
> loading should be done by adding to modules-load.d.
> 
> The oddity is that by loading as module is as follows (differences):
> 
> builtin iommu_v2 version dmesg:
> amdgpu: HMM registered 2048MB device memory
> amdgpu: Topology: Add APU node [0x0:0x0]
> amdgpu: Topology: Add APU node [0x15dd:0x1002]
> AMD-Vi: AMD IOMMUv2 loaded and initialized
> kfd kfd: amdgpu: added device 1002:15dd
> kfd kfd: amdgpu: Allocated 3969056 bytes on gart
> memmap_init_zone_device initialised 524288 pages in 0ms

IOMMU V2 modules provides IOMMU feature like attaching device to
process. I think amdgpu uses those features if available.
So in this case amdgpu is using those IOMMU features.

> 
> module not loaded due to missing iommu.conf dmesg:
> amdgpu: CRAT table disabled by module option
> amdgpu: Topology: Add CPU node
> amdgpu: Virtual CRAT table created for CPU
> kfd kfd: amdgpu: GC IP 090100 not supported in kfd
> 
> module load through iommu.conf dmesg:
> amdgpu: CRAT table disabled by module option
> amdgpu: Topology: Add CPU node
> amdgpu: Virtual CRAT table created for CPU
> AMD-Vi: AMD IOMMUv2 loaded and initialized
> kfd kfd: amdgpu: GC IP 090100 not supported in kfd
> 
> Note, only difference on witk/without iommu.conf is:
> AMD-Vi: AMD IOMMUv2 loaded and initialized

I think in this case iommu_v2.ko module got loaded after GPU
initialized. Hence amdgpu is not using iommu v2 features.


> 
> So does this mean missing features by not having builtin?
> If not, should Kconfig have hint about module and loading?

@Felix,
  I see that drivers/gpu/drm/amd/amdkfd/Kconfig contains below line
    imply AMD_IOMMU_V2 if X86_64


  Should we change `s/imply/select` ?

-Vasant


> 
> Steve
> 
> I wish to be personally CC'ed the answers/comments posted to the list
> in response to my posting, please:) Not a list member.
> 
> I hope you can assist linux people and myself. I assumed from dmesg that
> it must be builtin. But also wonder if it should be in amdgpu or tied to it.
> Steve
> 
>