[PATCH 2/2] drm/amdgpu: Let KFD use more VMIDs on Arcturus

[Christian König]

Am 25.06.20 um 17:58 schrieb Felix Kuehling:
> Am 2020-06-25 um 11:50 a.m. schrieb Christian König:
>> Am 25.06.20 um 17:43 schrieb Felix Kuehling:
>>> Am 2020-06-25 um 11:38 a.m. schrieb Christian König:
>>>> Am 25.06.20 um 17:15 schrieb Felix Kuehling:
>>>>> Am 2020-06-25 um 4:19 a.m. schrieb Christian König:
>>>>>> Am 25.06.20 um 05:18 schrieb Felix Kuehling:
>>>>>>> When there is no graphics support, KFD can use more of the VMIDs.
>>>>>>> Graphics
>>>>>>> VMIDs are only used for video decoding/encoding and post processing.
>>>>>>> With
>>>>>>> two VCE engines, there is no reason to reserve more than 2 VMIDs for
>>>>>>> that.
>>>>>> IIRC the expectation is that we still use the compute queues for post
>>>>>> processing and not the KFD.
>>>>>>
>>>>>> So we will need at least VMIDs for that as well.
>>>>> Correct. Post processing uses compute queues and VMIDs in the GFXHUB.
>>>>> VCE uses VMIDs in the MMHUB. I believe in Mesa they use the same VM
>>>>> context. So can't they share the same VMIDs?
>>>> Ah! Good point, But we still need at least 3 VMID when VMID
>>>> reservation is active.
>>> I don't know anything about that VMID reservation feature. What is it
>>> used for? Who is using it? How many VMIDs can be reserved?
>>>
>>> If one VMID is reserved, there would still be one VMID left for video
>>> post processing. That's not ideal, but I don't think it would be fatal.
>>> But is it a realistic use case that VMID reservation and ROCm+video
>>> processing would happen on the same system at the same time?
>> VMID reservation is used for debugging and yes there can only be one
>> reserved.
>>
>> But I think we need at least two for dynamic assignment or we might
>> run into a BUG_ON() while giving VMIDs to jobs.
> I don't see any BUGs or BUG_ONs in amdgpu_ids.c. What should I be
> looking out for?

We used to have a BUG_ON() when we couldn't find a VMID as alternative 
if the process already has one but needs to flush it.

But it's a long time ago that I last looked into this.

>> But I certainly need to test this as well. It's possible that 1 VMID
>> indeed works as expected.
> I could run the test, if you describe the problematic scenario you have
> in mind.

Just try to set the available VMIDs to 1 and see if GFX/Compute and MM 
submission work at the same time from multiple processes.

A few UVD video playbacks at the same time should do the job.

Regards,
Christian.

>
> Thanks,
>    Felix
>
>
>> Regards,
>> Christian.
>>
>>> Thanks,
>>>     Felix
>>>
>>>
>>>> I don't think you can go below this.
>>>>
>>>> Regards,
>>>> Christian.
>>>>
>>>>> Regards,
>>>>>      Felix
>>>>>
>>>>>
>>>>>>> Signed-off-by: Felix Kuehling <Felix.Kuehling at amd.com>
>>>>>>> ---
>>>>>>>      drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 11 ++++++++---
>>>>>>>      1 file changed, 8 insertions(+), 3 deletions(-)
>>>>>>>
>>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>> b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>> index 6e10b42c57e5..3470929e5b8e 100644
>>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>> @@ -1245,10 +1245,15 @@ static int gmc_v9_0_sw_init(void *handle)
>>>>>>>          /*
>>>>>>>           * number of VMs
>>>>>>>           * VMID 0 is reserved for System
>>>>>>> -     * amdgpu graphics/compute will use VMIDs 1-7
>>>>>>> -     * amdkfd will use VMIDs 8-15
>>>>>>> +     * amdgpu graphics/compute will use VMIDs 1..n-1
>>>>>>> +     * amdkfd will use VMIDs n..15
>>>>>>> +     *
>>>>>>> +     * The first KFD VMID is 8 for GPUs with graphics, 3 for
>>>>>>> +     * compute-only GPUs. On compute-only GPUs that leaves 2 VMIDs
>>>>>>> +     * for video processing.
>>>>>>>           */
>>>>>>> -    adev->vm_manager.first_kfd_vmid = 8;
>>>>>>> +    adev->vm_manager.first_kfd_vmid =
>>>>>>> +        adev->asic_type == CHIP_ARCTURUS ? 3 : 8;
>>>>>>>            amdgpu_vm_manager_init(adev);
>>>>>>>