[PATCH 1/1] amdgpu: Enable KFD on POWER systems

[Timothy Pearson]

----- Original Message -----
> From: "Felix Kuehling" <felix.kuehling at amd.com>
> To: "Timothy Pearson" <tpearson at raptorengineering.com>
> Cc: "amd-gfx" <amd-gfx at lists.freedesktop.org>
> Sent: Monday, November 25, 2019 3:34:20 PM
> Subject: Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems

> On 2019-11-25 4:06 p.m., Timothy Pearson wrote:
>>
>> ----- Original Message -----
>>> From: "Felix Kuehling" <felix.kuehling at amd.com>
>>> To: "Timothy Pearson" <tpearson at raptorengineering.com>, "amd-gfx"
>>> <amd-gfx at lists.freedesktop.org>
>>> Sent: Monday, November 25, 2019 11:07:31 AM
>>> Subject: Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
>>> Hi Timothy,
>>>
>>> Thank you for the patch and for confirming that it works. We did some
>>> experimental work on Power8 a few years ago. I see that Talos II is Power9.
>>>
>>> At the time we were working on Power8 we had to add some #ifdef
>>> CONFIG_ACPI guards around some ACPI-specific code in KFD. Do you know to
>>> what extent ACPI is available and working on the Power architecture?
>>>
>>> Another problem we ran into with Power, is the physical address map.
>>> System memory can be a physical addresses outside the range accessible
>>> by the GPU. Vega has 44-bit physical addressing. Older Polaris GPUs only
>>> have 40-bits. Did you run into any such problems? Do you need an IOMMU
>>> to make system memory accessible to the GPU?
>>>
>>> Regards,
>>>    Felix
>> Yes, we are POWER9.  It looks like the ACPI guards are no longer required; as
>> you have surmised, POWER does not use ACPI (the equivalent is OPAL, which is a
>> different interface entirely).  What were the APCI calls used for?  There may
>> be OPAL equivalents that could be added in to replace them and provide similar
>> functionality.
> 
> There are some ACPI calls (e.g. acpi_get_table) in kfd_crat.c for
> getting a CRAT table from ACPI. This is only useful for AMD APUs, which
> are x86_64. We don't need this for discrete GPUs because on non-APU
> systems there is no CRAT table and we build our own. If you can compile
> the code without problems on Power and with CONFIG_ACPI not defined,
> then I guess this is no longer an issue.

Sounds reasonable -- yes, it compiles without issue so I think we're good to go.

> 
>> Kernel 5.4 enables a > 32-bit and <=64-bit bypass mode for POWER.  This is one
>> reason we came back and revisited the KFD/ROCm functionality on POWER; as it
>> turns out, after fixing up the userspace tools KFD is indeed functional on
>> POWER with 5.4-rc8 and above.  My understanding is that the POWER IOMMU is used
>> as a lightweight translation layer between the 64-bit host and the 40/44-bit
>> GPU.
>>
>> I'm working on getting a Debian PPA set up for POWER to make the userspace tools
>> easier to obtain for testing, but progress is slow due to lack of Debian source
>> packages.  Probably the easiest way to replicate / test this with HIP is to use
>> the AOMP repository with my modifications; pull requests are already in place
>> on Github for most of the userspace tooling updates.
>>
>> Thank you!
> Thanks,
>   Felix
> 
> 
> _______________________________________________
> amd-gfx mailing list
> amd-gfx at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx