[PATCH 07/16] drm/amdgpu: add helper to create doorbell pages

[Shashank Sharma]

On 30/03/2023 16:15, Luben Tuikov wrote:
> On 2023-03-30 10:04, Shashank Sharma wrote:
>> On 30/03/2023 15:42, Luben Tuikov wrote:
>>> On 2023-03-29 11:47, Shashank Sharma wrote:
>>>> From: Shashank Sharma <contactshashanksharma at gmail.com>
>>>>
>>>> This patch adds helper functions to create and free doorbell
>>>> pages for kernel objects.
>>>>
>>>> Cc: Alex Deucher <alexander.deucher at amd.com>
>>>> Cc: Christian Koenig <christian.koenig at amd.com>
>>>> Signed-off-by: Shashank Sharma <shashank.sharma at amd.com>
>>>> ---
>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell.h  | 41 ++++++++++++++++
>>>>    .../gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c  | 49 +++++++++++++++++++
>>>>    2 files changed, 90 insertions(+)
>>>>
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell.h
>>>> index f9c3b77bf65d..6581b78fe438 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell.h
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell.h
>>>> @@ -27,6 +27,24 @@
>>>>    /*
>>>>     * GPU doorbell structures, functions & helpers
>>>>     */
>>>> +
>>>> +/* Structure to hold doorbell pages from PCI doorbell BAR */
>>>> +struct amdgpu_doorbell_obj {
>>> In the comment you say "Structure to hold ...";
>>> it is a C structure, but then in the name of a function we see "obj".
>>> (Object is something which is defined like in memory, i.e. it exists, not
>>> something which is only declared.)
>>> This is just a declaration of a structure, not an object per se.
>>> I'd call it "struct amdgpu_doorbell_struct" or just "struct amdgpu_doorbell".
>> It is similar to struct amdgpu buffer object (struct amdgpu_bo), and
>> many more existing structure.
> The amdpgu_bo is very different than a structure describing a doorbell.
> The doorbell description isn't really "an object". I understand
> the enthusiasm, but it is really not "an object". It's just a doorbell
> description. :-)

amdgpu_bo is page of ttm_memory with additional information,

amdgpu_doorbell_obj is a page of ttm_doorbells with additional information

(it is not just one doorbell description)

I don't see a problem here.

- Shashank

>
> Regards,
> Luben
>
>> - Shashank
>>
>>> Then in the definition, you can call it an object/objects, if you'd like,
>>> like "struct amdgpu_doorbell *doorb_object[];" then you can say
>>> "db = doorb_object[i]";
>>>
>>> Regards,
>>> Luben
>>>
>>>> +	struct amdgpu_bo *bo;
>>>> +	uint64_t gpu_addr;
>>>> +	uint32_t *cpu_addr;
>>>> +	uint32_t size;
>>>> +
>>>> +	/* First index in this object */
>>>> +	uint32_t start;
>>>> +
>>>> +	/* Last index in this object */
>>>> +	uint32_t end;
>>>> +
>>>> +	/* bitmap for dynamic doorbell allocation from this object */
>>>> +	unsigned long *doorbell_bitmap;
>>>> +};
>>>> +
>>>>    struct amdgpu_doorbell {
>>>>    	/* doorbell mmio */
>>>>    	resource_size_t		base;
>>>> @@ -328,6 +346,29 @@ int amdgpu_device_doorbell_init(struct amdgpu_device *adev);
>>>>     */
>>>>    void amdgpu_device_doorbell_fini(struct amdgpu_device *adev);
>>>>    
>>>> +/**
>>>> + * amdgpu_doorbell_free_page - Free a doorbell page
>>>> + *
>>>> + * @adev: amdgpu_device pointer
>>>> + *
>>>> + * @db_age: previously allocated doobell page details
>>>> + *
>>>> + */
>>>> +void amdgpu_doorbell_free_page(struct amdgpu_device *adev,
>>>> +				struct amdgpu_doorbell_obj *db_obj);
>>>> +
>>>> +/**
>>>> + * amdgpu_doorbell_alloc_page - create a page from doorbell pool
>>>> + *
>>>> + * @adev: amdgpu_device pointer
>>>> + *
>>>> + * @db_age: doobell page structure to fill details with
>>>> + *
>>>> + * returns 0 on success, else error number
>>>> + */
>>>> +int amdgpu_doorbell_alloc_page(struct amdgpu_device *adev,
>>>> +				struct amdgpu_doorbell_obj *db_obj);
>>>> +
>>>>    #define RDOORBELL32(index) amdgpu_mm_rdoorbell(adev, (index))
>>>>    #define WDOORBELL32(index, v) amdgpu_mm_wdoorbell(adev, (index), (v))
>>>>    #define RDOORBELL64(index) amdgpu_mm_rdoorbell64(adev, (index))
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c
>>>> index 1aea92363fd3..8be15b82b545 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c
>>>> @@ -111,6 +111,55 @@ void amdgpu_mm_wdoorbell64(struct amdgpu_device *adev, u32 index, u64 v)
>>>>    	}
>>>>    }
>>>>    
>>>> +/**
>>>> + * amdgpu_doorbell_free_page - Free a doorbell page
>>>> + *
>>>> + * @adev: amdgpu_device pointer
>>>> + *
>>>> + * @db_age: previously allocated doobell page details
>>>> + *
>>>> + */
>>>> +void amdgpu_doorbell_free_page(struct amdgpu_device *adev,
>>>> +					struct amdgpu_doorbell_obj *db_obj)
>>>> +{
>>>> +	amdgpu_bo_free_kernel(&db_obj->bo,
>>>> +			      &db_obj->gpu_addr,
>>>> +			      (void **)&db_obj->cpu_addr);
>>>> +
>>>> +}
>>>> +
>>>> +/**
>>>> + * amdgpu_doorbell_alloc_page - create a page from doorbell pool
>>>> + *
>>>> + * @adev: amdgpu_device pointer
>>>> + *
>>>> + * @db_age: doobell page structure to fill details with
>>>> + *
>>>> + * returns 0 on success, else error number
>>>> + */
>>>> +int amdgpu_doorbell_alloc_page(struct amdgpu_device *adev,
>>>> +				struct amdgpu_doorbell_obj *db_obj)
>>>> +{
>>>> +	int r;
>>>> +
>>>> +	db_obj->size = ALIGN(db_obj->size, PAGE_SIZE);
>>>> +
>>>> +	r = amdgpu_bo_create_kernel(adev,
>>>> +				    db_obj->size,
>>>> +				    PAGE_SIZE,
>>>> +				    AMDGPU_GEM_DOMAIN_DOORBELL,
>>>> +				    &db_obj->bo,
>>>> +				    &db_obj->gpu_addr,
>>>> +				    (void **)&db_obj->cpu_addr);
>>>> +
>>>> +	if (r) {
>>>> +		DRM_ERROR("Failed to create doorbell BO, err=%d\n", r);
>>>> +		return r;
>>>> +	}
>>>> +
>>>> +	return 0;
>>>> +}
>>>> +
>>>>    /*
>>>>     * GPU doorbell aperture helpers function.
>>>>     */