[PATCH 0/5] prevent OOM triggered by TTM

[Thomas Hellstrom]

Hi,

On 02/07/2018 02:22 PM, Christian König wrote:
>> Understood, but why is that?
> Well because customers requested it :)
>
> What we try to do here is having a parameter which says when less than 
> x megabytes of memory are left then fail the allocation.
>
> This is basically to prevent buggy applications which try to allocate 
> as much memory as possible until they receive an -ENOMEM from running 
> into the OOM killer.

OK. Understood.

>
>> That's true, but with VRAM, TTM overcommits swap space which may lead 
>> to ugly memory allocation failures at hibernate time.
> Yeah, that is exactly the reason why I said that Roger should disable 
> the limit during suspend swap out :)

Well that was really in the context of the swapping implementation 
rather in the context of this change so it was a little off-topic. Even 
if disabling this limit, TTM can overcommit. But looking at the swapping 
implementation is a different issue.

/Thomas





>
> Regards,
> Christian.
>
> Am 07.02.2018 um 14:17 schrieb Thomas Hellstrom:
>> Hi, Roger.
>>
>> On 02/07/2018 09:25 AM, He, Roger wrote:
>>>     Why should TTM be different in that aspect? It would be good to 
>>> know your reasoning WRT this?
>>>
>>> Now, in TTM struct ttm_bo_device it already has member no_retry to 
>>> indicate your option.
>>> If you prefer no OOM triggered by TTM, set it as true. The default 
>>> is false to keep original behavior.
>>> AMD prefers return value of no memory rather than OOM for now.
>>
>> Understood, but why is that? I mean just because TTM doesn't invoke 
>> the OOM killer, that doesn't mean that the process will, the next 
>> millisecond, page in a number of pages and invoke it? So this 
>> mechanism would be pretty susceptible to races?
>>>     One thing I looked at at one point was to have TTM do the 
>>> swapping itself instead of handing it off to the shmem system. That 
>>> way we could pre-allocate swap entries for all swappable (BO) 
>>> memory, making sure that we wouldn't run out of swap space when,
>>>
>>> I prefer current mechanism of swap out. At the beginning the swapped 
>>> pages stay in system memory by shmem until OS move to status with 
>>> high memory pressure, that has an obvious advantage. For example, if 
>>> the BO is swapped out into shmem, but not really be flushed into 
>>> swap disk. When validate it and swap in it at this moment, the 
>>> overhead is small compared to swap in from disk.
>>
>> But that is true for a page handed off to the swap-cache as well. It 
>> won't be immediately flushed to disc, only when the swap cache is 
>> shrunk.
>>
>>> In addition, No need swap space reservation for TTM pages when 
>>> allocation since swap disk is shared not only for TTM exclusive.
>>
>> That's true, but with VRAM, TTM overcommits swap space which may lead 
>> to ugly memory allocation failures at hibernate time.
>>
>>> So again we provide a flag no_retry in struct ttm_bo_device to let 
>>> driver set according to its request.
>>
>> Thanks,
>> Thomas
>>
>>
>>>
>>>
>>> Thanks
>>> Roger(Hongbo.He)
>>>
>>> -----Original Message-----
>>> From: Thomas Hellstrom [mailto:thomas at shipmail.org]
>>> Sent: Wednesday, February 07, 2018 2:43 PM
>>> To: He, Roger <Hongbo.He at amd.com>; amd-gfx at lists.freedesktop.org; 
>>> dri-devel at lists.freedesktop.org
>>> Cc: Koenig, Christian <Christian.Koenig at amd.com>
>>> Subject: Re: [PATCH 0/5] prevent OOM triggered by TTM
>>>
>>> Hi, Roger,
>>>
>>> On 02/06/2018 10:04 AM, Roger He wrote:
>>>> currently ttm code has no any allocation limit. So it allows pages
>>>> allocatation unlimited until OOM. Because if swap space is full of
>>>> swapped pages and then system memory will be filled up with ttm pages.
>>>> and then any memory allocation request will trigger OOM.
>>>>
>>> I'm a bit curious, isn't this the way things are supposed to work on 
>>> a linux system?
>>> If all memory resources are used up, the OOM killer will kill the 
>>> most memory hungry (perhaps rogue) process rather than processes 
>>> being nice and try to find out themselves whether allocations will 
>>> succeed?
>>> Why should TTM be different in that aspect? It would be good to know 
>>> your reasoning WRT this?
>>>
>>> Admittedly, graphics process OOM memory accounting doesn't work very 
>>> well, due to not all BOs not being CPU mapped, but it looks like 
>>> there is recent work towards fixing this?
>>>
>>> One thing I looked at at one point was to have TTM do the swapping 
>>> itself instead of handing it off to the shmem system. That way we 
>>> could pre-allocate swap entries for all swappable (BO) memory, 
>>> making sure that we wouldn't run out of swap space when, for 
>>> example, hibernating and that would also limit the pinned 
>>> non-swappable memory (from TTM driver kernel allocations for 
>>> example) to half the system memory resources.
>>>
>>> Thanks,
>>> Thomas
>>>
>>