[PATCH 2/2] drm/amdgpu: Let KFD use more VMIDs on Arcturus

[Felix Kuehling]

On 2020-06-25 12:01 p.m., Christian König wrote:
> Am 25.06.20 um 17:58 schrieb Felix Kuehling:
>> Am 2020-06-25 um 11:50 a.m. schrieb Christian König:
>>> Am 25.06.20 um 17:43 schrieb Felix Kuehling:
>>>> Am 2020-06-25 um 11:38 a.m. schrieb Christian König:
>>>>> Am 25.06.20 um 17:15 schrieb Felix Kuehling:
>>>>>> Am 2020-06-25 um 4:19 a.m. schrieb Christian König:
>>>>>>> Am 25.06.20 um 05:18 schrieb Felix Kuehling:
>>>>>>>> When there is no graphics support, KFD can use more of the VMIDs.
>>>>>>>> Graphics
>>>>>>>> VMIDs are only used for video decoding/encoding and post 
>>>>>>>> processing.
>>>>>>>> With
>>>>>>>> two VCE engines, there is no reason to reserve more than 2 
>>>>>>>> VMIDs for
>>>>>>>> that.
>>>>>>> IIRC the expectation is that we still use the compute queues for 
>>>>>>> post
>>>>>>> processing and not the KFD.
>>>>>>>
>>>>>>> So we will need at least VMIDs for that as well.
>>>>>> Correct. Post processing uses compute queues and VMIDs in the 
>>>>>> GFXHUB.
>>>>>> VCE uses VMIDs in the MMHUB. I believe in Mesa they use the same VM
>>>>>> context. So can't they share the same VMIDs?
>>>>> Ah! Good point, But we still need at least 3 VMID when VMID
>>>>> reservation is active.
>>>> I don't know anything about that VMID reservation feature. What is it
>>>> used for? Who is using it? How many VMIDs can be reserved?
>>>>
>>>> If one VMID is reserved, there would still be one VMID left for video
>>>> post processing. That's not ideal, but I don't think it would be 
>>>> fatal.
>>>> But is it a realistic use case that VMID reservation and ROCm+video
>>>> processing would happen on the same system at the same time?
>>> VMID reservation is used for debugging and yes there can only be one
>>> reserved.
>>>
>>> But I think we need at least two for dynamic assignment or we might
>>> run into a BUG_ON() while giving VMIDs to jobs.
>> I don't see any BUGs or BUG_ONs in amdgpu_ids.c. What should I be
>> looking out for?
>
> We used to have a BUG_ON() when we couldn't find a VMID as alternative 
> if the process already has one but needs to flush it.
>
> But it's a long time ago that I last looked into this.
>
>>> But I certainly need to test this as well. It's possible that 1 VMID
>>> indeed works as expected.
>> I could run the test, if you describe the problematic scenario you have
>> in mind.
>
> Just try to set the available VMIDs to 1 and see if GFX/Compute and MM 
> submission work at the same time from multiple processes.
>
> A few UVD video playbacks at the same time should do the job.

I tested it on Fiji with first_kfd_vmid=2, running 3 instances of VLC 
playing a 1080p h.264 video using VDPAU. (For some reason VA-API is 
broken: vaDeriveImage: operation failed). Just to be sure it was really 
using UVD, I disabled HW acceleration in VLC and saw CPU usage increase. 
Everything seems to be working fine.

Regards,
   Felix


>
> Regards,
> Christian.
>
>>
>> Thanks,
>>    Felix
>>
>>
>>> Regards,
>>> Christian.
>>>
>>>> Thanks,
>>>>     Felix
>>>>
>>>>
>>>>> I don't think you can go below this.
>>>>>
>>>>> Regards,
>>>>> Christian.
>>>>>
>>>>>> Regards,
>>>>>>      Felix
>>>>>>
>>>>>>
>>>>>>>> Signed-off-by: Felix Kuehling <Felix.Kuehling at amd.com>
>>>>>>>> ---
>>>>>>>>      drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 11 ++++++++---
>>>>>>>>      1 file changed, 8 insertions(+), 3 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>>> b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>>> index 6e10b42c57e5..3470929e5b8e 100644
>>>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>>> @@ -1245,10 +1245,15 @@ static int gmc_v9_0_sw_init(void *handle)
>>>>>>>>          /*
>>>>>>>>           * number of VMs
>>>>>>>>           * VMID 0 is reserved for System
>>>>>>>> -     * amdgpu graphics/compute will use VMIDs 1-7
>>>>>>>> -     * amdkfd will use VMIDs 8-15
>>>>>>>> +     * amdgpu graphics/compute will use VMIDs 1..n-1
>>>>>>>> +     * amdkfd will use VMIDs n..15
>>>>>>>> +     *
>>>>>>>> +     * The first KFD VMID is 8 for GPUs with graphics, 3 for
>>>>>>>> +     * compute-only GPUs. On compute-only GPUs that leaves 2 
>>>>>>>> VMIDs
>>>>>>>> +     * for video processing.
>>>>>>>>           */
>>>>>>>> -    adev->vm_manager.first_kfd_vmid = 8;
>>>>>>>> +    adev->vm_manager.first_kfd_vmid =
>>>>>>>> +        adev->asic_type == CHIP_ARCTURUS ? 3 : 8;
>>>>>>>>            amdgpu_vm_manager_init(adev);
>