[PATCH v9 13/14] drm/amdgpu: enable compute/gfx usermode queue
Sharma, Shashank
shashank.sharma at amd.com
Thu May 2 14:17:59 UTC 2024
On 02/05/2024 16:10, Alex Deucher wrote:
> On Thu, May 2, 2024 at 1:51 AM Sharma, Shashank <shashank.sharma at amd.com> wrote:
>>
>> On 01/05/2024 22:44, Alex Deucher wrote:
>>> On Fri, Apr 26, 2024 at 10:27 AM Shashank Sharma
>>> <shashank.sharma at amd.com> wrote:
>>>> From: Arvind Yadav <arvind.yadav at amd.com>
>>>>
>>>> This patch does the necessary changes required to
>>>> enable compute workload support using the existing
>>>> usermode queues infrastructure.
>>>>
>>>> Cc: Alex Deucher <alexander.deucher at amd.com>
>>>> Cc: Christian Koenig <christian.koenig at amd.com>
>>>> Signed-off-by: Arvind Yadav <arvind.yadav at amd.com>
>>>> Signed-off-by: Shashank Sharma <shashank.sharma at amd.com>
>>>> ---
>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c | 3 ++-
>>>> drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c | 2 ++
>>>> drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c | 10 +++++++++-
>>>> include/uapi/drm/amdgpu_drm.h | 1 +
>>>> 4 files changed, 14 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
>>>> index e516487e8db9..78d34fa7a0b9 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userqueue.c
>>>> @@ -189,7 +189,8 @@ amdgpu_userqueue_create(struct drm_file *filp, union drm_amdgpu_userq *args)
>>>> int qid, r = 0;
>>>>
>>>> /* Usermode queues are only supported for GFX/SDMA engines as of now */
>>>> - if (args->in.ip_type != AMDGPU_HW_IP_GFX && args->in.ip_type != AMDGPU_HW_IP_DMA) {
>>>> + if (args->in.ip_type != AMDGPU_HW_IP_GFX && args->in.ip_type != AMDGPU_HW_IP_DMA
>>>> + && args->in.ip_type != AMDGPU_HW_IP_COMPUTE) {
>>>> DRM_ERROR("Usermode queue doesn't support IP type %u\n", args->in.ip_type);
>>>> return -EINVAL;
>>>> }
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
>>>> index 525bd0f4d3f7..27b86f7fe949 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c
>>>> @@ -1350,6 +1350,7 @@ static int gfx_v11_0_sw_init(void *handle)
>>>> adev->gfx.mec.num_pipe_per_mec = 4;
>>>> adev->gfx.mec.num_queue_per_pipe = 4;
>>>> adev->userq_funcs[AMDGPU_HW_IP_GFX] = &userq_mes_v11_0_funcs;
>>>> + adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = &userq_mes_v11_0_funcs;
>>>> break;
>>>> case IP_VERSION(11, 0, 1):
>>>> case IP_VERSION(11, 0, 4):
>>>> @@ -1362,6 +1363,7 @@ static int gfx_v11_0_sw_init(void *handle)
>>>> adev->gfx.mec.num_pipe_per_mec = 4;
>>>> adev->gfx.mec.num_queue_per_pipe = 4;
>>>> adev->userq_funcs[AMDGPU_HW_IP_GFX] = &userq_mes_v11_0_funcs;
>>>> + adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = &userq_mes_v11_0_funcs;
>>>> break;
>>>> default:
>>>> adev->gfx.me.num_me = 1;
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
>>>> index a5e270eda37b..d61d80f86003 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/mes_v11_0_userqueue.c
>>>> @@ -183,7 +183,8 @@ static int mes_v11_0_userq_create_ctx_space(struct amdgpu_userq_mgr *uq_mgr,
>>>> }
>>>>
>>>> /* We don't need to set other FW objects for SDMA queues */
>>>> - if (queue->queue_type == AMDGPU_HW_IP_DMA)
>>>> + if ((queue->queue_type == AMDGPU_HW_IP_DMA) ||
>>>> + (queue->queue_type == AMDGPU_HW_IP_COMPUTE))
>>>> return 0;
>>>>
>>>> /* Shadow and GDS objects come directly from userspace */
>>>> @@ -246,6 +247,13 @@ static int mes_v11_0_userq_mqd_create(struct amdgpu_userq_mgr *uq_mgr,
>>>> userq_props->use_doorbell = true;
>>>> userq_props->doorbell_index = queue->doorbell_index;
>>>>
>>>> + if (queue->queue_type == AMDGPU_HW_IP_COMPUTE) {
>>>> + userq_props->eop_gpu_addr = mqd_user->eop_va;
>>>> + userq_props->hqd_pipe_priority = AMDGPU_GFX_PIPE_PRIO_NORMAL;
>>>> + userq_props->hqd_queue_priority = AMDGPU_GFX_QUEUE_PRIORITY_MINIMUM;
>>>> + userq_props->hqd_active = false;
>>>> + }
>>>> +
>>>> queue->userq_prop = userq_props;
>>>>
>>>> r = mqd_hw_default->init_mqd(adev, (void *)queue->mqd.cpu_ptr, userq_props);
>>>> diff --git a/include/uapi/drm/amdgpu_drm.h b/include/uapi/drm/amdgpu_drm.h
>>>> index 22f56a30f7cb..676792ad3618 100644
>>>> --- a/include/uapi/drm/amdgpu_drm.h
>>>> +++ b/include/uapi/drm/amdgpu_drm.h
>>>> @@ -375,6 +375,7 @@ struct drm_amdgpu_userq_mqd {
>>>> * sized.
>>>> */
>>>> __u64 csa_va;
>>>> + __u64 eop_va;
>>>> };
>>> Let's add a new mqd descriptor for compute since it's different from
>>> gfx and sdma.
>> the only different thing is this object (vs csa and gds objects), apart
>> from that, the mqd is the same as they all are MES based. Am I missing
>> something here ?
> The scheduling entity is irrelevant. The mqd is defined by the engine
> itself. E.g., v11_structs.h. Gfx has one set of requirements,
> compute has different ones, and SDMA has different ones. VPE and VCN
> also have mqds. When we add support for them in the future, they may
> have additional requirements. I want to make it clear in the
> interface what additional data are required for each ring type.
Yes, this comment was also with the first understanding, so please
ignore it.
We are aligned on the IP specific MQD structures now.
>
>>> Also, can we handle the eop buffer as part of the
>>> kernel metadata for compute user queues rather than having the user
>>> specify it?
>> Sure, we can do it.
> Thinking about it more, I think the eop has to be in the user's GPU
> virtual address space so it probably makes more sense for the user to
> allocate this, but ideally we'd take an extra ref count on it while
> the queue is active to avoid the user freeing it while the queue is
> active, but that can probably be a future improvement.
I was also thinking if the BO is expected to be created by (VMID != 0),
so keeping it in userspace makes it aligned with other IP specific MQD
objects.
Lets keep it for userspace, but will create a separate Compute MQD object .
- Shashank
> Alex
>
>> - Shashank
>>
>>> Alex
>>>
>>>> struct drm_amdgpu_userq_in {
>>>> --
>>>> 2.43.2
>>>>
More information about the amd-gfx
mailing list