[PATCH 07/25] drm/amdgpu: Update kgd2kfd_shared_resources for dGPU support

[Felix Kuehling]

On 2018-01-27 04:19 AM, Christian König wrote:
> Am 27.01.2018 um 02:09 schrieb Felix Kuehling:
>> Add GPUVM size and DRM render node. Also add function to query the
>> VMID mask to avoid hard-coding it in multiple places later.
>>
>> Signed-off-by: Felix Kuehling <Felix.Kuehling at amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c      | 19
>> +++++++++++++++++--
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h      |  2 ++
>>   drivers/gpu/drm/amd/include/kgd_kfd_interface.h |  6 ++++++
>>   3 files changed, 25 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
>> index c9f204d..294c467 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
>> @@ -30,6 +30,8 @@
>>   const struct kgd2kfd_calls *kgd2kfd;
>>   bool (*kgd2kfd_init_p)(unsigned int, const struct kgd2kfd_calls**);
>>   +static const unsigned int compute_vmid_bitmap = 0xFF00;
>> +
>>   int amdgpu_amdkfd_init(void)
>>   {
>>       int ret;
>> @@ -137,9 +139,12 @@ void amdgpu_amdkfd_device_init(struct
>> amdgpu_device *adev)
>>       int last_valid_bit;
>>       if (adev->kfd) {
>>           struct kgd2kfd_shared_resources gpu_resources = {
>> -            .compute_vmid_bitmap = 0xFF00,
>> +            .compute_vmid_bitmap = compute_vmid_bitmap,
>>               .num_pipe_per_mec = adev->gfx.mec.num_pipe_per_mec,
>> -            .num_queue_per_pipe = adev->gfx.mec.num_queue_per_pipe
>> +            .num_queue_per_pipe = adev->gfx.mec.num_queue_per_pipe,
>> +            .gpuvm_size = adev->vm_manager.max_pfn
>> +                        << AMDGPU_GPU_PAGE_SHIFT,
>
> That most likely doesn't work as intended on Vega10. The address space
> is divided into an upper and a lower range, but max_pfn includes both.
>
> I suggest to use something like min(adev->vm_manager.max_pfn <<
> AMDGPU_GPU_PAGE_SHIFT, AMDGPU_VM_HOLE_START).

I think this is fine as it is. This just tells the Thunk the size of the
virtual address space supported by the GPU. Currently the Thunk only
uses 40-bits for SVM. But eventually it will be able to use the entire
47 bits of user address space. Any excess address space will just go unused.

I'm also wondering how universal the split 48-bit virtual address space 
layout is. Even for x86_64 there seems to be a 5-level page table layout
that supports 56 bits of user mode addresses
(Documentation/x86/x86_64/mm.txt). AArch64 seems to support 48-bit user
mode addresses (Documentation/arm64/memory.txt). I haven't found similar
information for PowerPC yet.

We should avoid coding too much architecture-specific logic into this
driver that's supposed to support other architectures as well. I should
also review the aperture placement with bigger user mode address spaces
in mind.

Regards,
  Felix

>
> Christian.
>
>> +            .drm_render_minor = adev->ddev->render->index
>>           };
>>             /* this is going to have a few of the MSBs set that we
>> need to
>> @@ -351,3 +356,13 @@ uint64_t amdgpu_amdkfd_get_vram_usage(struct
>> kgd_dev *kgd)
>>         return amdgpu_vram_mgr_usage(&adev->mman.bdev.man[TTM_PL_VRAM]);
>>   }
>> +
>> +bool amdgpu_amdkfd_is_kfd_vmid(struct amdgpu_device *adev, u32 vmid)
>> +{
>> +    if (adev->kfd) {
>> +        if ((1 << vmid) & compute_vmid_bitmap)
>> +            return true;
>> +    }
>> +
>> +    return false;
>> +}
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
>> index 8d92f5c..cc3aa13 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
>> @@ -66,6 +66,8 @@ void amdgpu_amdkfd_device_fini(struct amdgpu_device
>> *adev);
>>   struct kfd2kgd_calls *amdgpu_amdkfd_gfx_7_get_functions(void);
>>   struct kfd2kgd_calls *amdgpu_amdkfd_gfx_8_0_get_functions(void);
>>   +bool amdgpu_amdkfd_is_kfd_vmid(struct amdgpu_device *adev, u32 vmid);
>> +
>>   /* Shared API */
>>   int alloc_gtt_mem(struct kgd_dev *kgd, size_t size,
>>               void **mem_obj, uint64_t *gpu_addr,
>> diff --git a/drivers/gpu/drm/amd/include/kgd_kfd_interface.h
>> b/drivers/gpu/drm/amd/include/kgd_kfd_interface.h
>> index 9e35249..36c706a 100644
>> --- a/drivers/gpu/drm/amd/include/kgd_kfd_interface.h
>> +++ b/drivers/gpu/drm/amd/include/kgd_kfd_interface.h
>> @@ -108,6 +108,12 @@ struct kgd2kfd_shared_resources {
>>         /* Number of bytes at start of aperture reserved for KGD. */
>>       size_t doorbell_start_offset;
>> +
>> +    /* GPUVM address space size in bytes */
>> +    uint64_t gpuvm_size;
>> +
>> +    /* Minor device number of the render node */
>> +    int drm_render_minor;
>>   };
>>     struct tile_config {
>