Documentation about AMD's HSA implementation?

[Felix Kuehling]

On 2018-02-13 04:06 PM, Ming Yang wrote:
> Thanks for the suggestions!  But I might ask several specific
> questions, as I can't find the answer in those documents, to give
> myself a quick start if that's okay. Pointing me to the
> files/functions would be good enough.  Any explanations are
> appreciated.   My purpose is to hack it with different scheduling
> policy with real-time and predictability consideration.
>
> - Where/How is the packet scheduler implemented?  How are packets from
> multiple queues scheduled?  What about scheduling packets from queues
> in different address spaces?

This is done mostly in firmware. The CP engine supports up to 32 queues.
We share those between KFD and AMDGPU. KFD gets 24 queues to use.
Usually that is 6 queues times 4 pipes. Pipes are threads in the CP
micro engine. Within each pipe the queues are time-multiplexed.

If we need more than 24 queues, or if we have more than 8 processes, the
hardware scheduler (HWS) adds another layer scheduling, basically
round-robin between batches of 24 queues or 8 processes. Once you get
into such an over-subscribed scenario your performance and GPU
utilization can suffers quite badly.

>
> - I noticed the new support of concurrency of multi-processes in the
> archive of this mailing list.  Could you point me to the code that
> implements this?

That's basically just a switch that tells the firmware that it is
allowed to schedule queues from different processes at the same time.
The upper limit is the number of VMIDs that HWS can work with. It needs
to assign a unique VMID to each process (each VMID representing a
separate address space, page table, etc.). If there are more processes
than VMIDs, the HWS has to time-multiplex.

>
> - Also another related question -- where/how is the preemption/context
> switch between packets/queues implemented?

As long as you don't oversubscribe the available VMIDs, there is no real
context switching. Everything can run concurrently. When you start
oversubscribing HW queues or VMIDs, the HWS firmware will start
multiplexing. This is all handled inside the firmware and is quite
transparent even to KFD.

KFD interacts with the HWS firmware through the HIQ (HSA interface
queue). It supports packets for unmapping queues, we can send it a new
runlist (basically a bunch of map-process and map-queue packets). The
interesting files to look at are kfd_packet_manager.c,
kfd_kernel_queue_<hw>.c and kfd_device_queue_manager.c.

Regards,
  Felix

>
> Thanks in advance!
>
> Best,
> Mark
>
>> On 13 Feb 2018, at 2:56 PM, Felix Kuehling <felix.kuehling at amd.com> wrote:
>> There is also this: https://gpuopen.com/professional-compute/, which
>> give pointer to several libraries and tools that built on top of ROCm.
>>
>> Another thing to keep in mind is, that ROCm is diverging from the strict
>> HSA standard in some important ways. For example the HSA standard
>> includes HSAIL as an intermediate representation that gets finalized on
>> the target system, whereas ROCm compiles directly to native GPU ISA.
>>
>> Regards,
>>   Felix
>>
>> On Tue, Feb 13, 2018 at 9:40 AM, Deucher, Alexander <Alexander.Deucher at amd.com> wrote:
>>> The ROCm documentation is probably a good place to start:
>>>
>>> https://rocm.github.io/documentation.html
>>>
>>>
>>> Alex
>>>
>>> ________________________________
>>> From: amd-gfx <amd-gfx-bounces at lists.freedesktop.org> on behalf of Ming Yang
>>> <minos.future at gmail.com>
>>> Sent: Tuesday, February 13, 2018 12:00 AM
>>> To: amd-gfx at lists.freedesktop.org
>>> Subject: Documentation about AMD's HSA implementation?
>>>
>>> Hi,
>>>
>>> I'm interested in HSA and excited when I found AMD's fully open-stack ROCm
>>> supporting it. Before digging into the code, I wonder if there's any
>>> documentation available about AMD's HSA implementation, either book,
>>> whitepaper, paper, or documentation.
>>>
>>> I did find helpful materials about HSA, including HSA standards on this page
>>> (http://www.hsafoundation.com/standards/) and a nice book about HSA
>>> (Heterogeneous System Architecture A New Compute Platform Infrastructure).
>>> But regarding the documentation about AMD's implementation, I haven't found
>>> anything yet.
>>>
>>> Please let me know if there are ones publicly accessible. If no, any
>>> suggestions on learning the implementation of specific system components,
>>> e.g., queue scheduling.
>>>
>>> Best,
>>> Mark