<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style> </head> <body dir="ltr"> <p style="font-family:Arial;font-size:10pt;color:#317100;margin:15pt;" align="Left"> [AMD Public Use]<br> </p> <br> <div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> Maybe we should re-test the problematic piglit test and if it's no longer an issue, revert:</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <span><br> </span></div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <span>commit 24562523688bebc7ec17a88271b4e8c3fc337b74<br> </span> <div>Author: Andrey Grodzovsky <andrey.grodzovsky@amd.com><br> </div> <div>Date: Fri Dec 15 12:09:16 2017 -0500<br> </div> <div><br> </div> <div> Revert "drm/amd/amdgpu: set gtt size according to system memory size only"<br> </div> <div> <br> </div> <div> This reverts commit ba851eed895c76be0eb4260bdbeb7e26f9ccfaa2.<br> </div> <div> With that change piglit max size tests (running with -t max.*size) are causing<br> </div> <div> OOM and hard hang on my CZ with 1GB RAM.<br> </div> <div> <br> </div> <div> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com><br> </div> <div> Acked-by: Alex Deucher <alexander.deucher@amd.com><br> </div> <div> Reviewed-by: Christian König <christian.koenig@amd.com><br> </div> <div> Reviewed-by: Roger He <Hongbo.He@amd.com><br> </div> <div> Signed-off-by: Alex Deucher <alexander.deucher@amd.com><br> </div> <span></span><br> </div> <div id="appendonsend"></div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> amd-gfx <amd-gfx-bounces@lists.freedesktop.org> on behalf of Christian König <ckoenig.leichtzumerken@gmail.com><br> <b>Sent:</b> Wednesday, July 15, 2020 5:28 AM<br> <b>To:</b> Kuehling, Felix <Felix.Kuehling@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; amd-gfx list <amd-gfx@lists.freedesktop.org><br> <b>Subject:</b> Re: Failed to find memory space for buffer eviction</font> <div> </div> </div> <div class="BodyFragment"><font size="2"><span style="font-size:11pt;"> <div class="PlainText">Am 15.07.20 um 04:49 schrieb Felix Kuehling:<br> > Am 2020-07-14 um 4:28 a.m. schrieb Christian König:<br> >> Hi Felix,<br> >><br> >> yes I already stumbled over this as well quite recently.<br> >><br> >> See the following patch which I pushed to drm-misc-next just yesterday:<br> >><br> >> commit e04be2310b5eac683ec03b096c0e22c4c2e23593<br> >> Author: Christian König <christian.koenig@amd.com><br> >> Date: Mon Jul 6 17:32:55 2020 +0200<br> >><br> >> drm/ttm: further cleanup ttm_mem_reg handling<br> >><br> >> Stop touching the backend private pointer alltogether and<br> >> make sure we never put the same mem twice by.<br> >><br> >> Signed-off-by: Christian König <christian.koenig@amd.com><br> >> Reviewed-by: Madhav Chauhan <madhav.chauhan@amd.com><br> >> Link: <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchwork.freedesktop.org%2Fpatch%2F375613%2F&data=02%7C01%7Calexander.deucher%40amd.com%7Ce19192b295fc41a7fb4c08d828a168d4%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637304021017509156&sdata=zilZiBrs%2FVrzhZuolVzhLSO2kIBDugp16HT58G7tX8w%3D&reserved=0"> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchwork.freedesktop.org%2Fpatch%2F375613%2F&data=02%7C01%7Calexander.deucher%40amd.com%7Ce19192b295fc41a7fb4c08d828a168d4%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637304021017509156&sdata=zilZiBrs%2FVrzhZuolVzhLSO2kIBDugp16HT58G7tX8w%3D&reserved=0</a><br> >><br> >><br> >> But this shouldn't have been problematic since we used a dummy value<br> >> for mem->mm_node in this case.<br> > Hmm, yeah, I was reading the code wrong. It's possible that I was really<br> > just out of GTT space. But see below.<br> <br> It looks like it yes.<br> <br> >> What could be problematic and result is an overrun is that TTM was<br> >> buggy and called put_node twice for the same memory.<br> >><br> >> So I've seen that the code needs fixing as well, but I'm not 100% sure<br> >> how you ran into your problem.<br> > This is in the KFD eviction test, which deliberately overcommits VRAM in<br> > order to trigger lots of evictions. It will use some GTT space while BOs<br> > are evicted. But shouldn't it move them further out of GTT and into<br> > SYSTEM to free up GTT space?<br> <br> Yes, exactly that should happen.<br> <br> But for some reason it couldn't find a candidate to evict and the 14371 <br> pages left are just a bit to small for the buffer.<br> <br> Regards,<br> Christian.<br> <br> > Your change "further cleanup ttm_mem_reg handling" removes a<br> > mem->mm_node = NULL in ttm_bo_handle_move_mem in exactly the case where<br> > a BO is moved from GTT to SYSTEM. I think that leads to a later put_node<br> > call not happening or amdgpu_gtt_mgr_del returning before incrementing<br> > mgr->available.<br> ><br> > I can try if cherry-picking your two fixes will help with the eviction test.<br> ><br> > Regards,<br> > Felix<br> ><br> ><br> >> Regards,<br> >> Christian.<br> >><br> >> Am 14.07.20 um 02:44 schrieb Felix Kuehling:<br> >>> I'm running into this problem with the KFD EvictionTest. The log snippet<br> >>> below looks like it ran out of GTT space for the eviction of a 64MB<br> >>> buffer. But then it dumps the used and free space and shows plenty of<br> >>> free space.<br> >>><br> >>> As I understand it, the per-page breakdown of used and free space shown<br> >>> by TTM is the GART space. So it's not very meaningful.<br> >>><br> >>> What matters more is the GTT space managed by amdgpu_gtt_mgr.c. And<br> >>> that's where the problem is. It keeps track of available GTT space with<br> >>> an atomic counter in amdgpu_gtt_mgr.available. It gets decremented in<br> >>> amdgpu_gtt_mgr_new and incremented in amdgpu_gtt_mgr_del. The trouble<br> >>> is, that TTM doesn't call the latter for ttm_mem_regs that don't have an<br> >>> mm_node:<br> >>><br> >>>> void ttm_bo_mem_put(struct ttm_buffer_object *bo, struct ttm_mem_reg<br> >>>> *mem)<br> >>>> {<br> >>>> struct ttm_mem_type_manager *man =<br> >>>> &bo->bdev->man[mem->mem_type];<br> >>>><br> >>>> if (mem->mm_node)<br> >>>> (*man->func->put_node)(man, mem);<br> >>>> }<br> >>> GTT BOs that don't have GART space allocated, don't hate an mm_node. So<br> >>> the amdgpu_gtt_mgr.available counter doesn't get incremented when an<br> >>> unmapped GTT BO is freed, and eventually runs out of space.<br> >>><br> >>> Now I know what the problem is, but I don't know how to fix it. Maybe a<br> >>> dummy-mm_node for unmapped GTT BOs, to trick TTM into calling our<br> >>> put_node callback? Or a change in TTM to call put_node unconditionally?<br> >>><br> >>> Regards,<br> >>> Felix<br> >>><br> >>><br> >>> [ 360.082552] [TTM] Failed to find memory space for buffer<br> >>> 0x00000000264c823c eviction<br> >>> [ 360.090331] [TTM] No space for 00000000264c823c (16384 pages,<br> >>> 65536K, 64M)<br> >>> [ 360.090334] [TTM] placement[0]=0x00010002 (1)<br> >>> [ 360.090336] [TTM] has_type: 1<br> >>> [ 360.090337] [TTM] use_type: 1<br> >>> [ 360.090339] [TTM] flags: 0x0000000A<br> >>> [ 360.090341] [TTM] gpu_offset: 0xFF00000000<br> >>> [ 360.090342] [TTM] size: 1048576<br> >>> [ 360.090344] [TTM] available_caching: 0x00070000<br> >>> [ 360.090346] [TTM] default_caching: 0x00010000<br> >>> [ 360.090349] [TTM] 0x0000000000000400-0x0000000000000402: 2: used<br> >>> [ 360.090352] [TTM] 0x0000000000000402-0x0000000000000404: 2: used<br> >>> [ 360.090354] [TTM] 0x0000000000000404-0x0000000000000406: 2: used<br> >>> [ 360.090355] [TTM] 0x0000000000000406-0x0000000000000408: 2: used<br> >>> [ 360.090357] [TTM] 0x0000000000000408-0x000000000000040a: 2: used<br> >>> [ 360.090359] [TTM] 0x000000000000040a-0x000000000000040c: 2: used<br> >>> [ 360.090361] [TTM] 0x000000000000040c-0x000000000000040e: 2: used<br> >>> [ 360.090363] [TTM] 0x000000000000040e-0x0000000000000410: 2: used<br> >>> [ 360.090365] [TTM] 0x0000000000000410-0x0000000000000412: 2: used<br> >>> [ 360.090367] [TTM] 0x0000000000000412-0x0000000000000414: 2: used<br> >>> [ 360.090368] [TTM] 0x0000000000000414-0x0000000000000415: 1: used<br> >>> [ 360.090370] [TTM] 0x0000000000000415-0x0000000000000515: 256: used<br> >>> [ 360.090372] [TTM] 0x0000000000000515-0x0000000000000516: 1: used<br> >>> [ 360.090374] [TTM] 0x0000000000000516-0x0000000000000517: 1: used<br> >>> [ 360.090376] [TTM] 0x0000000000000517-0x0000000000000518: 1: used<br> >>> [ 360.090378] [TTM] 0x0000000000000518-0x0000000000000519: 1: used<br> >>> [ 360.090379] [TTM] 0x0000000000000519-0x000000000000051a: 1: used<br> >>> [ 360.090381] [TTM] 0x000000000000051a-0x000000000000051b: 1: used<br> >>> [ 360.090383] [TTM] 0x000000000000051b-0x000000000000051c: 1: used<br> >>> [ 360.090385] [TTM] 0x000000000000051c-0x000000000000051d: 1: used<br> >>> [ 360.090387] [TTM] 0x000000000000051d-0x000000000000051f: 2: used<br> >>> [ 360.090389] [TTM] 0x000000000000051f-0x0000000000000521: 2: used<br> >>> [ 360.090391] [TTM] 0x0000000000000521-0x0000000000000522: 1: used<br> >>> [ 360.090392] [TTM] 0x0000000000000522-0x0000000000000523: 1: used<br> >>> [ 360.090394] [TTM] 0x0000000000000523-0x0000000000000524: 1: used<br> >>> [ 360.090396] [TTM] 0x0000000000000524-0x0000000000000525: 1: used<br> >>> [ 360.090398] [TTM] 0x0000000000000525-0x0000000000000625: 256: used<br> >>> [ 360.090400] [TTM] 0x0000000000000625-0x0000000000000725: 256: used<br> >>> [ 360.090402] [TTM] 0x0000000000000725-0x0000000000000727: 2: used<br> >>> [ 360.090404] [TTM] 0x0000000000000727-0x00000000000007c0: 153: used<br> >>> [ 360.090406] [TTM] 0x00000000000007c0-0x0000000000000b8a: 970: used<br> >>> [ 360.090407] [TTM] 0x0000000000000b8a-0x0000000000000b8b: 1: used<br> >>> [ 360.090409] [TTM] 0x0000000000000b8b-0x0000000000000bcb: 64: used<br> >>> [ 360.090411] [TTM] 0x0000000000000bcb-0x0000000000000bcd: 2: used<br> >>> [ 360.090413] [TTM] 0x0000000000000bcd-0x0000000000040000: 259123:<br> >>> free<br> >>> [ 360.090415] [TTM] total: 261120, used 1997 free 259123<br> >>> [ 360.090417] [TTM] man size:1048576 pages, gtt available:14371 pages,<br> >>> usage:4039MB<br> >>><br> >>><br> > _______________________________________________<br> > amd-gfx mailing list<br> > amd-gfx@lists.freedesktop.org<br> > <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Calexander.deucher%40amd.com%7Ce19192b295fc41a7fb4c08d828a168d4%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637304021017509156&sdata=LVRisNun0DYM%2F5dLthnxNiN0KgAq%2BAh5mXvnoYEjkR0%3D&reserved=0"> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Calexander.deucher%40amd.com%7Ce19192b295fc41a7fb4c08d828a168d4%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637304021017509156&sdata=LVRisNun0DYM%2F5dLthnxNiN0KgAq%2BAh5mXvnoYEjkR0%3D&reserved=0</a><br> <br> _______________________________________________<br> amd-gfx mailing list<br> amd-gfx@lists.freedesktop.org<br> <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Calexander.deucher%40amd.com%7Ce19192b295fc41a7fb4c08d828a168d4%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637304021017509156&sdata=LVRisNun0DYM%2F5dLthnxNiN0KgAq%2BAh5mXvnoYEjkR0%3D&reserved=0">https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Calexander.deucher%40amd.com%7Ce19192b295fc41a7fb4c08d828a168d4%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637304021017509156&sdata=LVRisNun0DYM%2F5dLthnxNiN0KgAq%2BAh5mXvnoYEjkR0%3D&reserved=0</a><br> </div> </span></font></div> </div> </body> </html>