<div dir="auto">Christian, should we set this flag for GDS too? Will it help with GDS OOM failures?<div dir="auto"><br></div><div dir="auto">Marek</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri., May 13, 2022, 07:26 Christian König, <<a href="mailto:ckoenig.leichtzumerken@gmail.com">ckoenig.leichtzumerken@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Exactly that's what we can't do.<br>
<br>
See the kernel must always be able to move things to GTT or discard. So <br>
when you want to guarantee that something is in VRAM you must at the <br>
same time say you can discard it if it can't.<br>
<br>
Christian.<br>
<br>
Am 13.05.22 um 10:43 schrieb Pierre-Eric Pelloux-Prayer:<br>
> Hi Marek, Christian,<br>
><br>
> If the main feature for Mesa of AMDGPU_GEM_CREATE_DISCARDABLE is <br>
> getting the best placement, maybe we should have 2 separate flags:<br>
> * AMDGPU_GEM_CREATE_DISCARDABLE: indicates to the kernel that it can <br>
> discards the content on eviction instead of preserving it<br>
> * AMDGPU_GEM_CREATE_FORCE_BEST_PLACEMENT (or <br>
> AMDGPU_GEM_CREATE_NO_GTT_FALLBACK ? or AMDGPU_CREATE_GEM_AVOID_GTT?): <br>
> tells the kernel that this bo really needs to be in VRAM<br>
><br>
><br>
> Pierre-Eric<br>
><br>
> On 13/05/2022 00:17, Marek Olšák wrote:<br>
>> Would it be better to set the VM_ALWAYS_VALID flag to have a greater <br>
>> guarantee that the best placement will be chosen?<br>
>><br>
>> See, the main feature is getting the best placement, not being <br>
>> discardable. The best placement is a hw design requirement due to <br>
>> using memory for uses that are expected to have performance similar <br>
>> to onchip SRAMs. We need to make sure the best placement is <br>
>> guaranteed if it's VRAM.<br>
>><br>
>> Marek<br>
>><br>
>> On Thu., May 12, 2022, 03:26 Christian König, <br>
>> <<a href="mailto:ckoenig.leichtzumerken@gmail.com" target="_blank" rel="noreferrer">ckoenig.leichtzumerken@gmail.com</a> <br>
>> <mailto:<a href="mailto:ckoenig.leichtzumerken@gmail.com" target="_blank" rel="noreferrer">ckoenig.leichtzumerken@gmail.com</a>>> wrote:<br>
>><br>
>> Am 12.05.22 um 00:06 schrieb Marek Olšák:<br>
>>> 3rd question: Is it worth using this on APUs?<br>
>><br>
>> It makes memory management somewhat easier when we are really OOM.<br>
>><br>
>> E.g. it should also work for GTT allocations and when the core <br>
>> kernel says "Hey please free something up or I will start the <br>
>> OOM-killer" it's something we can easily throw away.<br>
>><br>
>> Not sure how many of those buffers we have, but marking <br>
>> everything which is temporary with that flag is probably a good idea.<br>
>><br>
>>><br>
>>> Thanks,<br>
>>> Marek<br>
>>><br>
>>> On Wed, May 11, 2022 at 5:58 PM Marek Olšák <<a href="mailto:maraeo@gmail.com" target="_blank" rel="noreferrer">maraeo@gmail.com</a> <br>
>>> <mailto:<a href="mailto:maraeo@gmail.com" target="_blank" rel="noreferrer">maraeo@gmail.com</a>>> wrote:<br>
>>><br>
>>> Will the kernel keep all discardable buffers in VRAM if VRAM <br>
>>> is not overcommitted by discardable buffers, or will other buffers <br>
>>> also affect the placement of discardable buffers?<br>
>>><br>
>><br>
>> Regarding the eviction pressure the buffers will be handled like <br>
>> any other buffer, but instead of preserving the content it is just <br>
>> discarded on eviction.<br>
>><br>
>>><br>
>>> Do evictions deallocate the buffer, or do they keep an <br>
>>> allocation in GTT and only the copy is skipped?<br>
>>><br>
>><br>
>> It really deallocates the backing store of the buffer, just keeps <br>
>> a dummy page array around where all entries are NULL.<br>
>><br>
>> There is a patch set on the mailing list to make this a little <br>
>> bit more efficient, but even using the dummy page array should only <br>
>> have a few bytes overhead.<br>
>><br>
>> Regards,<br>
>> Christian.<br>
>><br>
>>><br>
>>> Thanks,<br>
>>> Marek<br>
>>><br>
>>> On Wed, May 11, 2022 at 3:08 AM Marek Olšák <br>
>>> <<a href="mailto:maraeo@gmail.com" target="_blank" rel="noreferrer">maraeo@gmail.com</a> <mailto:<a href="mailto:maraeo@gmail.com" target="_blank" rel="noreferrer">maraeo@gmail.com</a>>> wrote:<br>
>>><br>
>>> OK that sounds good.<br>
>>><br>
>>> Marek<br>
>>><br>
>>> On Wed, May 11, 2022 at 2:04 AM Christian König <br>
>>> <<a href="mailto:ckoenig.leichtzumerken@gmail.com" target="_blank" rel="noreferrer">ckoenig.leichtzumerken@gmail.com</a> <br>
>>> <mailto:<a href="mailto:ckoenig.leichtzumerken@gmail.com" target="_blank" rel="noreferrer">ckoenig.leichtzumerken@gmail.com</a>>> wrote:<br>
>>><br>
>>> Hi Marek,<br>
>>><br>
>>> Am 10.05.22 um 22:43 schrieb Marek Olšák:<br>
>>>> A better flag name would be:<br>
>>>> AMDGPU_GEM_CREATE_BEST_PLACEMENT_OR_DISCARD<br>
>>><br>
>>> A bit long for my taste and I think the best <br>
>>> placement is just a side effect.<br>
>>><br>
>>>><br>
>>>> Marek<br>
>>>><br>
>>>> On Tue, May 10, 2022 at 4:13 PM Marek Olšák <br>
>>>> <<a href="mailto:maraeo@gmail.com" target="_blank" rel="noreferrer">maraeo@gmail.com</a> <mailto:<a href="mailto:maraeo@gmail.com" target="_blank" rel="noreferrer">maraeo@gmail.com</a>>> wrote:<br>
>>>><br>
>>>> Does this really guarantee VRAM placement? The <br>
>>>> code doesn't say anything about that.<br>
>>>><br>
>>><br>
>>> Yes, see the code here:<br>
>>><br>
>>>><br>
>>>> diff --git <br>
>>>> a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c <br>
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c<br>
>>>> index 8b7ee1142d9a..1944ef37a61e 100644<br>
>>>> --- <br>
>>>> a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c<br>
>>>> +++ <br>
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c<br>
>>>> @@ -567,6 +567,7 @@ int <br>
>>>> amdgpu_bo_create(struct amdgpu_device *adev,<br>
>>>> bp->domain;<br>
>>>> bo->allowed_domains = <br>
>>>> bo->preferred_domains;<br>
>>>> if (bp->type != ttm_bo_type_kernel &&<br>
>>>> + !(bp->flags & <br>
>>>> AMDGPU_GEM_CREATE_DISCARDABLE) &&<br>
>>>> bo->allowed_domains == <br>
>>>> AMDGPU_GEM_DOMAIN_VRAM)<br>
>>>> bo->allowed_domains |= AMDGPU_GEM_DOMAIN_GTT;<br>
>>>><br>
>>><br>
>>> The only case where this could be circumvented is <br>
>>> when you try to allocate more than physically available on an APU.<br>
>>><br>
>>> E.g. you only have something like 32 MiB VRAM and <br>
>>> request 64 MiB, then the GEM code will catch the error and fallback <br>
>>> to GTT (IIRC).<br>
>>><br>
>>> Regards,<br>
>>> Christian.<br>
>>><br>
>><br>
<br>
</blockquote></div>