[PATCH 2/2] drm/amdgpu: Let KFD use more VMIDs on Arcturus

[Christian König]

Am 25.06.20 um 22:39 schrieb Felix Kuehling:
>
> On 2020-06-25 12:01 p.m., Christian König wrote:
>> Am 25.06.20 um 17:58 schrieb Felix Kuehling:
>>> Am 2020-06-25 um 11:50 a.m. schrieb Christian König:
>>>> Am 25.06.20 um 17:43 schrieb Felix Kuehling:
>>>>> Am 2020-06-25 um 11:38 a.m. schrieb Christian König:
>>>>>> Am 25.06.20 um 17:15 schrieb Felix Kuehling:
>>>>>>> Am 2020-06-25 um 4:19 a.m. schrieb Christian König:
>>>>>>>> Am 25.06.20 um 05:18 schrieb Felix Kuehling:
>>>>>>>>> When there is no graphics support, KFD can use more of the VMIDs.
>>>>>>>>> Graphics
>>>>>>>>> VMIDs are only used for video decoding/encoding and post 
>>>>>>>>> processing.
>>>>>>>>> With
>>>>>>>>> two VCE engines, there is no reason to reserve more than 2 
>>>>>>>>> VMIDs for
>>>>>>>>> that.
>>>>>>>> IIRC the expectation is that we still use the compute queues 
>>>>>>>> for post
>>>>>>>> processing and not the KFD.
>>>>>>>>
>>>>>>>> So we will need at least VMIDs for that as well.
>>>>>>> Correct. Post processing uses compute queues and VMIDs in the 
>>>>>>> GFXHUB.
>>>>>>> VCE uses VMIDs in the MMHUB. I believe in Mesa they use the same VM
>>>>>>> context. So can't they share the same VMIDs?
>>>>>> Ah! Good point, But we still need at least 3 VMID when VMID
>>>>>> reservation is active.
>>>>> I don't know anything about that VMID reservation feature. What is it
>>>>> used for? Who is using it? How many VMIDs can be reserved?
>>>>>
>>>>> If one VMID is reserved, there would still be one VMID left for video
>>>>> post processing. That's not ideal, but I don't think it would be 
>>>>> fatal.
>>>>> But is it a realistic use case that VMID reservation and ROCm+video
>>>>> processing would happen on the same system at the same time?
>>>> VMID reservation is used for debugging and yes there can only be one
>>>> reserved.
>>>>
>>>> But I think we need at least two for dynamic assignment or we might
>>>> run into a BUG_ON() while giving VMIDs to jobs.
>>> I don't see any BUGs or BUG_ONs in amdgpu_ids.c. What should I be
>>> looking out for?
>>
>> We used to have a BUG_ON() when we couldn't find a VMID as 
>> alternative if the process already has one but needs to flush it.
>>
>> But it's a long time ago that I last looked into this.
>>
>>>> But I certainly need to test this as well. It's possible that 1 VMID
>>>> indeed works as expected.
>>> I could run the test, if you describe the problematic scenario you have
>>> in mind.
>>
>> Just try to set the available VMIDs to 1 and see if GFX/Compute and 
>> MM submission work at the same time from multiple processes.
>>
>> A few UVD video playbacks at the same time should do the job.
>
> I tested it on Fiji with first_kfd_vmid=2, running 3 instances of VLC 
> playing a 1080p h.264 video using VDPAU. (For some reason VA-API is 
> broken: vaDeriveImage: operation failed). Just to be sure it was 
> really using UVD, I disabled HW acceleration in VLC and saw CPU usage 
> increase. Everything seems to be working fine.

In this case the patch is Reviewed-by: Christian König 
<christian.koenig at amd.com>

>
> Regards,
>   Felix
>
>
>>
>> Regards,
>> Christian.
>>
>>>
>>> Thanks,
>>>    Felix
>>>
>>>
>>>> Regards,
>>>> Christian.
>>>>
>>>>> Thanks,
>>>>>     Felix
>>>>>
>>>>>
>>>>>> I don't think you can go below this.
>>>>>>
>>>>>> Regards,
>>>>>> Christian.
>>>>>>
>>>>>>> Regards,
>>>>>>>      Felix
>>>>>>>
>>>>>>>
>>>>>>>>> Signed-off-by: Felix Kuehling <Felix.Kuehling at amd.com>
>>>>>>>>> ---
>>>>>>>>>      drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 11 ++++++++---
>>>>>>>>>      1 file changed, 8 insertions(+), 3 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>>>> b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>>>> index 6e10b42c57e5..3470929e5b8e 100644
>>>>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
>>>>>>>>> @@ -1245,10 +1245,15 @@ static int gmc_v9_0_sw_init(void *handle)
>>>>>>>>>          /*
>>>>>>>>>           * number of VMs
>>>>>>>>>           * VMID 0 is reserved for System
>>>>>>>>> -     * amdgpu graphics/compute will use VMIDs 1-7
>>>>>>>>> -     * amdkfd will use VMIDs 8-15
>>>>>>>>> +     * amdgpu graphics/compute will use VMIDs 1..n-1
>>>>>>>>> +     * amdkfd will use VMIDs n..15
>>>>>>>>> +     *
>>>>>>>>> +     * The first KFD VMID is 8 for GPUs with graphics, 3 for
>>>>>>>>> +     * compute-only GPUs. On compute-only GPUs that leaves 2 
>>>>>>>>> VMIDs
>>>>>>>>> +     * for video processing.
>>>>>>>>>           */
>>>>>>>>> -    adev->vm_manager.first_kfd_vmid = 8;
>>>>>>>>> +    adev->vm_manager.first_kfd_vmid =
>>>>>>>>> +        adev->asic_type == CHIP_ARCTURUS ? 3 : 8;
>>>>>>>>>            amdgpu_vm_manager_init(adev);
>>