[PATCH] drm/amdgpu: only use kernel zone if need_dma32 is not required

[Koenig, Christian]

Am 13.06.19 um 18:37 schrieb Kuehling, Felix:
> [SNIP]
>>>> There is
>>>> an issue found by KFDTest.BigBufStressTest, it allocates buffers up to
>>>> 3/8 of total 256GB system memory, each buffer size is 128MB, then use
>>>> queue to write to the buffers. If ttm_mem_global_alloc_page get page pfn
>>>> is below 4GB, it account to dma32 zone and will exhaust 2GB limit, then
>>>> ttm_check_swapping will schedule ttm_shrink_work to start eviction. It
>>>> takes minutes to finish restore (retry many times if busy), the test
>>>> failed because queue timeout. This eviction is unnecessary because we
>>>> still have enough free system memory.
>>> No that is definitely necessary. For example on my Laptop it's the sound
>>> system which needs DMA32.
>>>
>>> Without this when an application uses a lot of memory we run into the
>>> OOM killer as soon as some audio starts playing.
>>>
>> I did not realize TTM is used by other drivers. I agree we cannot remove
>> dma32 zone, this will break other device drivers which depends on dma32
>> zone.
> If I understand Christian correctly, the point is not that other drivers
> use TTM, but other drivers need dma32 memory (memory with 32-bit
> physical addresses). If TTM used up all that memory, it would break
> those other drivers. As a good steward of memory, TTM limits its usage
> of dma32 memory in order to allow other drivers to have access to it.

Yes, exactly.

>
> Regards,
>     Felix
>
>
>>>> It's random case, happens about 1/5. I can change test to increase the
>>>> timeout value to workaround this, but this seems TTM bug. This will slow
>>>> application performance a lot if this random issue happens.
>>> One thing we could try is to dig into why the kernel gives us DMA32
>>> pages when there are still other pages available. Please take a look at
>>> /proc/buddyinfo on that box for this.
>>>
>> Thanks for the advise, from buddyinfo, the machine has 4 nodes, each
>> node has 64GB memory, and dma32 zone is on node 0. BigBufStressTest
>> allocate 96GB. If I force the test on node 1, "numactl --cpunodebind=1
>> ./kfdtest", no eviction at all. If I force the test on node 0, it always
>> has eviction and restore because it used up all memory including dma32
>> zone from node 0. I will change test app to avoid running on node 0 to
>> workaround this.

That is a really interesting test case you got here.

I actually think that exhausting DMA32 before looking into another numa 
node is a bug in the core MM.

Anyway I will put it on my TODO list to improve the handling in TTM, 
shouldn't be more than a day or two of work.

Till that's done the workaround of not using node 0 should do it.

Christian.

>>
>> Thanks,
>> Philip
>>