[PATCH] drm/ttm: Fix the memory delay free issue

Koenig, Christian Christian.Koenig at amd.com
Mon Jul 15 10:54:22 UTC 2019


Hi Emily,

no, we can only cleanup the current one because we don't have a 
reference to the other ones.

At least that's how I understand you question,
Christian.

Am 15.07.19 um 12:47 schrieb Deng, Emily:
> Hi Christian,
>       Do you think we could free all those bos those are in current destroy list when the current resv is signal in ttm_bo_cleanup_refs?
>
> Best wishes
> Emily Deng
>
>> -----Original Message-----
>> From: Koenig, Christian <Christian.Koenig at amd.com>
>> Sent: Monday, July 15, 2019 5:41 PM
>> To: Deng, Emily <Emily.Deng at amd.com>; Zhou, David(ChunMing)
>> <David1.Zhou at amd.com>
>> Cc: amd-gfx at lists.freedesktop.org
>> Subject: Re: [PATCH] drm/ttm: Fix the memory delay free issue
>>
>>> Do you think we don't need to fix it?
>> No, when the application is exhausting memory then we can't expect anything
>> else here.
>>
>> See memory freeing is always delayed until it isn't used any more or when the
>> process is killed after access is prevented (by clearing page tables for example).
>>
>> What we could do is maybe look into why we don't block until the memory is
>> freed during command submission, but apart from that this sounds like
>> perfectly expected behavior.
>>
>> Regards,
>> Christian.
>>
>> Am 15.07.19 um 11:34 schrieb Deng, Emily:
>>> Hi Christian,
>>>       As has this behavior, when test vulkan cts allocation test, it will
>> exhausting the memory, and cause out of memory. Do you think we don't
>> need to fix it?
>>> Best wishes
>>> Emily Deng
>>>> -----Original Message-----
>>>> From: Koenig, Christian <Christian.Koenig at amd.com>
>>>> Sent: Monday, July 15, 2019 5:31 PM
>>>> To: Deng, Emily <Emily.Deng at amd.com>; Zhou, David(ChunMing)
>>>> <David1.Zhou at amd.com>
>>>> Cc: amd-gfx at lists.freedesktop.org
>>>> Subject: Re: [PATCH] drm/ttm: Fix the memory delay free issue
>>>>
>>>> Hi guys,
>>>>
>>>>> Do you have any suggestion about this? For per vm bo, it seems
>>>>> always
>>>> delay to free the ttm bo.
>>>> Yeah, and that is correct behavior.
>>>>
>>>> Since we don't know who is using a per-vm BO we need to wait for all
>>>> command submissions in flight when it is freed.
>>>>
>>>> For this we copy the current state of the shared reservation object
>>>> into the private one in ttm_bo_individualize_resv.
>>>>
>>>> Regards,
>>>> Christian.
>>>>
>>>> Am 15.07.19 um 08:49 schrieb Deng, Emily:
>>>>> Hi David,
>>>>>         You are right, it will copy per-vm resv.
>>>>>         But currently, it still has the delay free issue which non
>>>>> per vm bo doesn't
>>>> has. Maybe it already has new fences append to this resv object before
>> copy.
>>>>> Hi Christian,
>>>>>        Do you have any suggestion about this? For per vm bo, it seems
>>>>> always
>>>> delay to free the ttm bo.
>>>>> Best wishes
>>>>> Emily Deng
>>>>>> -----Original Message-----
>>>>>> From: Zhou, David(ChunMing) <David1.Zhou at amd.com>
>>>>>> Sent: Wednesday, July 10, 2019 9:28 PM
>>>>>> To: Deng, Emily <Emily.Deng at amd.com>; amd-
>> gfx at lists.freedesktop.org
>>>>>> Subject: Re: [PATCH] drm/ttm: Fix the memory delay free issue
>>>>>>
>>>>>> It doesn't make sense that freeing BO still uses per-vm resv.
>>>>>>
>>>>>> I remember when BO is in release list, its resv will be from per-vm resv
>> copy.
>>>>>> Could you check it?
>>>>>>
>>>>>> -David
>>>>>>
>>>>>> 在 2019/7/10 17:29, Emily Deng 写道:
>>>>>>> For vulkan cts allocation test cases, they will create a series of
>>>>>>> bos, and then free them. As it has lots of alloction test cases
>>>>>>> with the same vm, as per vm bo feature enable, all of those bos'
>>>>>>> resv are the same. But the bo free is quite slow, as they use the
>>>>>>> same resv object, for every time, free a bo, it will check the
>>>>>>> resv whether signal, if it signal, then will free it. But as the
>>>>>>> test cases will continue to create bo, and the resv fence is
>>>>>>> increasing. So the free is more
>>>>>> slower than creating. It will cause memory exhausting.
>>>>>>> Method:
>>>>>>> When the resv signal, release all the bos which are use the same
>>>>>>> resv object.
>>>>>>>
>>>>>>> Signed-off-by: Emily Deng <Emily.Deng at amd.com>
>>>>>>> ---
>>>>>>>      drivers/gpu/drm/ttm/ttm_bo.c | 29 ++++++++++++++++++++++++----
>> -
>>>>>>>      1 file changed, 24 insertions(+), 5 deletions(-)
>>>>>>>
>>>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c
>>>>>>> b/drivers/gpu/drm/ttm/ttm_bo.c index f9a3d4c..57ec59b 100644
>>>>>>> --- a/drivers/gpu/drm/ttm/ttm_bo.c
>>>>>>> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
>>>>>>> @@ -543,6 +543,7 @@ static int ttm_bo_cleanup_refs(struct
>>>>>> ttm_buffer_object *bo,
>>>>>>>      {
>>>>>>>      	struct ttm_bo_global *glob = bo->bdev->glob;
>>>>>>>      	struct reservation_object *resv;
>>>>>>> +	struct ttm_buffer_object *resv_bo, *resv_bo_next;
>>>>>>>      	int ret;
>>>>>>>
>>>>>>>      	if (unlikely(list_empty(&bo->ddestroy)))
>>>>>>> @@ -566,10 +567,14 @@ static int ttm_bo_cleanup_refs(struct
>>>>>> ttm_buffer_object *bo,
>>>>>>>      							   interruptible,
>>>>>>>      							   30 * HZ);
>>>>>>>
>>>>>>> -		if (lret < 0)
>>>>>>> +		if (lret < 0) {
>>>>>>> +			kref_put(&bo->list_kref, ttm_bo_release_list);
>>>>>>>      			return lret;
>>>>>>> -		else if (lret == 0)
>>>>>>> +		}
>>>>>>> +		else if (lret == 0) {
>>>>>>> +			kref_put(&bo->list_kref, ttm_bo_release_list);
>>>>>>>      			return -EBUSY;
>>>>>>> +		}
>>>>>>>
>>>>>>>      		spin_lock(&glob->lru_lock);
>>>>>>>      		if (unlock_resv &&
>>>>>>> !kcl_reservation_object_trylock(bo->resv))
>>>>>> { @@
>>>>>>> -582,6 +587,7 @@ static int ttm_bo_cleanup_refs(struct
>>>>>>> ttm_buffer_object
>>>>>> *bo,
>>>>>>>      			 * here.
>>>>>>>      			 */
>>>>>>>      			spin_unlock(&glob->lru_lock);
>>>>>>> +			kref_put(&bo->list_kref, ttm_bo_release_list);
>>>>>>>      			return 0;
>>>>>>>      		}
>>>>>>>      		ret = 0;
>>>>>>> @@ -591,15 +597,29 @@ static int ttm_bo_cleanup_refs(struct
>>>>>> ttm_buffer_object *bo,
>>>>>>>      		if (unlock_resv)
>>>>>>>      			kcl_reservation_object_unlock(bo->resv);
>>>>>>>      		spin_unlock(&glob->lru_lock);
>>>>>>> +		kref_put(&bo->list_kref, ttm_bo_release_list);
>>>>>>>      		return ret;
>>>>>>>      	}
>>>>>>>
>>>>>>>      	ttm_bo_del_from_lru(bo);
>>>>>>>      	list_del_init(&bo->ddestroy);
>>>>>>>      	kref_put(&bo->list_kref, ttm_bo_ref_bug);
>>>>>>> -
>>>>>>>      	spin_unlock(&glob->lru_lock);
>>>>>>>      	ttm_bo_cleanup_memtype_use(bo);
>>>>>>> +	kref_put(&bo->list_kref, ttm_bo_release_list);
>>>>>>> +
>>>>>>> +	spin_lock(&glob->lru_lock);
>>>>>>> +	list_for_each_entry_safe(resv_bo, resv_bo_next, &bo->bdev-
>>>>>>> ddestroy, ddestroy) {
>>>>>>> +		if (resv_bo->resv == bo->resv) {
>>>>>>> +			ttm_bo_del_from_lru(resv_bo);
>>>>>>> +			list_del_init(&resv_bo->ddestroy);
>>>>>>> +			spin_unlock(&glob->lru_lock);
>>>>>>> +			ttm_bo_cleanup_memtype_use(resv_bo);
>>>>>>> +			kref_put(&resv_bo->list_kref,
>> ttm_bo_release_list);
>>>>>>> +			spin_lock(&glob->lru_lock);
>>>>>>> +		}
>>>>>>> +	}
>>>>>>> +	spin_unlock(&glob->lru_lock);
>>>>>>>
>>>>>>>      	if (unlock_resv)
>>>>>>>      		kcl_reservation_object_unlock(bo->resv);
>>>>>>> @@ -639,9 +659,8 @@ static bool ttm_bo_delayed_delete(struct
>>>>>> ttm_bo_device *bdev, bool remove_all)
>>>>>>>      			ttm_bo_cleanup_refs(bo, false, !remove_all,
>> true);
>>>>>>>      		} else {
>>>>>>>      			spin_unlock(&glob->lru_lock);
>>>>>>> +			kref_put(&bo->list_kref, ttm_bo_release_list);
>>>>>>>      		}
>>>>>>> -
>>>>>>> -		kref_put(&bo->list_kref, ttm_bo_release_list);
>>>>>>>      		spin_lock(&glob->lru_lock);
>>>>>>>      	}
>>>>>>>      	list_splice_tail(&removed, &bdev->ddestroy);



More information about the amd-gfx mailing list