[PATCH 0/9] Replace use of radeon_sa with a new sub allocator

[Christian König]

Am 31.12.2014 um 14:39 schrieb Oded Gabbay:
> Background:
>
> amdkfd needs GART memory for several things, such as runlist packets,
> MQDs, HPDs and more. Unfortunately, all of this memory must be always
> pinned (due to several reasons which were discussed during the
> initial review of amdkfd).

In general seems to be a good idea, but so far I still don't have seen a 
good explanation why all those memory must be pinned. So please 
summarize that one once more.

Regards,
Christian.

>
> Current Solution:
>
> The current (short/mid-term) solution that was proposed by Jerome.G, is
> to limit the amount of memory to a small size, roughly 4MB and allocate
> this buffer at the start of the GART. To accomodate this, amdkfd has
> two kernel module parameters, maximum number of HSA processes and
> maximum number of queues per process, which require under 4MB of GART
> memory when using their defaults, 32 and 128 respectively.
>
> Until now, amdkfd used the radeon sub-allocator module (radeon_sa)
> to handle the sub-allocation of memory from this large buffer to
> different modules inside the amdkfd.
>
> However, while running OpenCL conformance test suite, we found that
> radeon_sa module is not suitable for this kind of task, due to its
> design:
> 1. Every allocation increments its interal pointer so the next
> allocation is *always* done ahead of the previous allocation. This
> causes the internal pointer to wrap-around when it reaches the end of
> the buffer.
>
> 2. When encoutering an area that is already allocated, the module
> waits for that area to be freed. If it is not freed in a timely manner
> (or has no fence), the allocation fails. Simply put, it can't "skip"
> the allocated area.
>
> Now, this is most probably good for graphics, but for amdkfd needs,
> the combination of the two behaviors mentioned above eventually causes
> a denial-of-service. This is because some memory allocations
> are *always* present and *never* freed (such as HPDs).
> Therefore, given enough time and workload, the radeon_sa eventually
> wraps around, encounters an already allocated area and gets stuck.
>
> Proposed new solution:
>
> To solve this, I have written a simple sub-allocator module inside
> amdkfd. It allocates fixed-size contiguous chunks (1 or more) and uses
> a bitmap to manage the allocations. The next allocation is always
> being searched for from the start of the GART buffer, and the module
> knows how to skip allocated chunks.
>
> Because most allocations are MQDs, and MQDs are 512 Bytes in size, I
> set the default chunk size to be 512 Bytes.
>
> The basic GART memory allocation is still being done in the
> amdkfd <--> radeon interface, and it still occupies less than 4MB.
>
> I have chosen to implement a new allocator instead of changing
> radeon_sa because the behavior of radeon_sa is very appropriate for
> graphics, where allocations do not stay forever. Also, amdkfd doesn't
> actually need the flexibility and features radeon_sa provides.
>
> 	Oded
>
> Oded Gabbay (9):
>    drm/amd: Add new kfd-->kgd interface for gart usage
>    drm/radeon: Impl. new gtt allocate/free functions
>    drm/amdkfd: Add gtt sa related data to kfd_dev struct
>    drm/amdkfd: Add kfd gtt sub-allocator functions
>    drm/amdkfd: Fixed calculation of gart buffer size
>    drm/amdkfd: Allocate gart memory using new interface
>    drm/amdkfd: Using new gtt sa in amdkfd
>    drm/radeon: Remove old radeon_sa usage from kfd-->kgd interface
>    drm/amd: Remove old radeon_sa funcs from kfd-->kgd interface
>
>   drivers/gpu/drm/amd/amdkfd/kfd_device.c            | 217 ++++++++++++++++++++-
>   .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c  |  23 +--
>   drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c      |  41 ++--
>   drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager.c       |  16 +-
>   drivers/gpu/drm/amd/amdkfd/kfd_packet_manager.c    |  10 +-
>   drivers/gpu/drm/amd/amdkfd/kfd_priv.h              |  28 ++-
>   drivers/gpu/drm/amd/include/kgd_kfd_interface.h    |  23 +--
>   drivers/gpu/drm/radeon/radeon_kfd.c                | 128 ++++++------
>   8 files changed, 329 insertions(+), 157 deletions(-)
>