[PATCH] drm/amdkfd: Fix partial migrate issue

[Felix Kuehling]

On 2025-01-06 21:10, Deng, Emily wrote:
> [AMD Official Use Only - AMD Internal Distribution Only]
>
>> -----Original Message-----
>> From: Kuehling, Felix <Felix.Kuehling at amd.com>
>> Sent: Tuesday, January 7, 2025 4:53 AM
>> To: Deng, Emily <Emily.Deng at amd.com>; amd-gfx at lists.freedesktop.org
>> Subject: Re: [PATCH] drm/amdkfd: Fix partial migrate issue
>>
>>
>> On 2025-01-05 20:45, Deng, Emily wrote:
>>> [AMD Official Use Only - AMD Internal Distribution Only]
>>>
>>>> -----Original Message-----
>>>> From: Kuehling, Felix <Felix.Kuehling at amd.com>
>>>> Sent: Saturday, January 4, 2025 7:10 AM
>>>> To: Deng, Emily <Emily.Deng at amd.com>; amd-gfx at lists.freedesktop.org
>>>> Subject: Re: [PATCH] drm/amdkfd: Fix partial migrate issue
>>>>
>>>>
>>>> On 2025-01-02 19:06, Emily Deng wrote:
>>>>> For partial migrate from ram to vram, the migrate->cpages is not
>>>>> equal to migrate->npages, should use migrate->npages to check all
>>>>> needed migrate pages which could be copied or not.
>>>>>
>>>>> And only need to set those pages could be migrated to
>>>>> migrate->dst[i], or the migrate_vma_pages will migrate the wrong
>>>>> pages based on the migrate- dst[i].
>>>>>
>>>>> Signed-off-by: Emily Deng <Emily.Deng at amd.com>
>>>> Good catch. But I think it's still not quite right. See inline.
>>>>
>>>>
>>>>> ---
>>>>>     drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 8 ++++----
>>>>>     1 file changed, 4 insertions(+), 4 deletions(-)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
>>>>> b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
>>>>> index 4b275937d05e..5c96c2d425e3 100644
>>>>> --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
>>>>> +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
>>>>> @@ -278,7 +278,7 @@ svm_migrate_copy_to_vram(struct kfd_node *node,
>>>>> struct
>>>> svm_range *prange,
>>>>>                         struct migrate_vma *migrate, struct dma_fence **mfence,
>>>>>                         dma_addr_t *scratch, uint64_t ttm_res_offset)
>>>>>     {
>>>>> -    uint64_t npages = migrate->cpages;
>>>>> +    uint64_t npages = migrate->npages;
>>>>>        struct amdgpu_device *adev = node->adev;
>>>>>        struct device *dev = adev->dev;
>>>>>        struct amdgpu_res_cursor cursor; @@ -299,9 +299,6 @@
>>>>> svm_migrate_copy_to_vram(struct kfd_node *node, struct
>>>> svm_range *prange,
>>>>>                struct page *spage;
>>>>>
>>>>>                dst[i] = cursor.start + (j << PAGE_SHIFT);
>>>>> -            migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]);
>>>>> -            svm_migrate_get_vram_page(prange, migrate->dst[i]);
>>>>> -            migrate->dst[i] = migrate_pfn(migrate->dst[i]);
>>>>>
>>>>>                spage = migrate_pfn_to_page(migrate->src[i]);
>>>>>                if (spage && !is_zone_device_page(spage)) {
>>>> We should also check (migrate->src[i] & MIGRATE_PFN_MIGRATE) to catch
>>>> only the pages that are ready to migrate (i.e. not already in device
>>>> memory and not pinned by someone else).
>>>>
>>>> Regards,
>>>>     Felix
>>>> The code " if (spage && !is_zone_device_page(spage)) {" is already checked
>> whether the page could be migrated or not. Because after called migrate_vma_setup,
>> for those pages couldn't be migrated it will be set to NULL.
>>
>> The page may be non-NULL. If the MIGRATE_PFN_MIGRATE is not set, the page
>> could not be collected for migration by migrate_vma_setup and it should not be
>> migrated. See
>> https://elixir.bootlin.com/linux/v6.12.6/source/mm/migrate_device.c#L470
>>
>> Regards,
>>    Felix
> Sorry, don't understand clearly. If a page couldn't be migrated, the src should be set NULL, if not, I think it is the migrate_vma_collect's bug. So could you point which code path
> it will not be NULL.

Looks like I misread the documentation. I thought there could be cases 
where a valid page would have the MIGRATE_PFN_MIGRATE flag unset. But I 
can't find that in the code in mm/migrate_device.c. In the cases where a 
page cannot be collected, the src entry will be 0. The only case I see 
where the MIGRATE_PFN_MIGRATE flag is significant is in cases where host 
memory was not allocated yet (PFN is 0) but the migration can allocate 
0-initialized device memory for it.

Regards,
   Felix


>>> Emily Deng
>>> Best Wishes
>>>
>>>
>>>>> @@ -345,6 +342,9 @@ svm_migrate_copy_to_vram(struct kfd_node *node,
>>>>> struct
>>>> svm_range *prange,
>>>>>                } else {
>>>>>                        j++;
>>>>>                }
>>>>> +            migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]);
>>>>> +            svm_migrate_get_vram_page(prange, migrate->dst[i]);
>>>>> +            migrate->dst[i] = migrate_pfn(migrate->dst[i]);
>>>>>        }
>>>>>
>>>>>        r = svm_migrate_copy_memory_gart(adev, src + i - j, dst + i -
>>>>> j, j,