Random short freezes due to TTM buffer migrations

Wed Aug 17 02:13:11 UTC 2016

It becomes regular for application to request a big chunk of memory and do the sub-allocation by itself. 
I agree Kernel should do better to provide fine grained paging granularity. 
I don't know whether 1/2 of vram is the biggest single allocation application would like to do, but it is better if we can remove this limitation in the future as well.

Thanks. 
Best Regards,
David

-----Original Message-----
From: Zhou, David(ChunMing) 
Sent: Wednesday, August 17, 2016 9:58 AM
To: Zhou, David(ChunMing) <David1.Zhou at amd.com>; Kuehling, Felix <Felix.Kuehling at amd.com>; Christian König <deathsimple at vodafone.de>; Marek Olšák <maraeo at gmail.com>; amd-gfx at lists.freedesktop.org; Mao, David <David.Mao at amd.com>
Subject: RE: Random short freezes due to TTM buffer migrations

Add his email.

> -----Original Message-----
> From: amd-gfx [mailto:amd-gfx-bounces at lists.freedesktop.org] On Behalf 
> Of Zhou, David(ChunMing)
> Sent: Wednesday, August 17, 2016 9:57 AM
> To: Kuehling, Felix <Felix.Kuehling at amd.com>; Christian König 
> <deathsimple at vodafone.de>; Marek Olšák <maraeo at gmail.com>; amd- 
> gfx at lists.freedesktop.org
> Subject: RE: Random short freezes due to TTM buffer migrations
> 
> +David Mao,
> 
> Well, our Vulcan stack aslo encountered this problem before, the 
> performance is very low when migration is often. At that moment, we 
> want to add some algorithm for eviction LRU, but failed to find an 
> appropriate generic  way. Then UMD decreased some VRAM usage at last.
> Hope we can get a solution for full VRAM usage this time.
> 
> Regards,
> David Zhou
> 
> > -----Original Message-----
> > From: amd-gfx [mailto:amd-gfx-bounces at lists.freedesktop.org] On 
> > Behalf Of Felix Kuehling
> > Sent: Wednesday, August 17, 2016 2:34 AM
> > To: Christian König <deathsimple at vodafone.de>; Marek Olšák 
> > <maraeo at gmail.com>; amd-gfx at lists.freedesktop.org
> > Subject: Re: Random short freezes due to TTM buffer migrations
> >
> > Very nice. I'm looking forward to this for KFD as well.
> >
> > One question: Will it be possible to share these split BOs as dmabufs?
> >
> > Regards,
> >   Felix
> >
> >
> > On 16-08-16 11:27 AM, Christian König wrote:
> > > Hi Marek,
> > >
> > > I'm already working on this.
> > >
> > > My current approach is to use a custom BO manager for VRAM with 
> > > TTM and so split allocations into chunks of 4MB.
> > >
> > > Large BOs are still swapped out as one, but it makes it much more 
> > > likely to that you can allocate 1/2 of VRAM as one buffer.
> > >
> > > Give me till the end of the week to finish this and then we can 
> > > test if that's sufficient or if we need to do more.
> > >
> > > Regards,
> > > Christian.
> > >
> > > Am 16.08.2016 um 16:33 schrieb Marek Olšák:
> > >> Hi,
> > >>
> > >> I'm seeing random temporary freezes (up to 2 seconds) under 
> > >> memory pressure. Before I describe the exact circumstances, I'd 
> > >> like to say that this is a serious issue affecting playability of 
> > >> certain AAA Linux games.
> > >>
> > >> In order to reproduce this, an application should:
> > >> - allocate a few very large buffers (256-512 MB per buffer)
> > >> - allocate more memory than there is available VRAM. The issue 
> > >> also occurs (but at a lower frequency) if the app needs only 80% of VRAM.
> > >>
> > >> Example: ttm_bo_validate needs to migrate a 512 MB buffer. The 
> > >> total size of moved memory for that call can be as high as 1.5 GB.
> > >> This is always followed by a big temporary drop in VRAM usage.
> > >>
> > >> The game I'm testing needs 3.4 GB of VRAM.
> > >>
> > >> Setups:
> > >> Tonga - 2 GB: It's nearly unplayable, because freezes occur too often.
> > >> Fiji - 4 GB: There is one freeze at the beginning (which is 
> > >> annoying too), after that it's smooth.
> > >>
> > >> So even 4 GB is not enough.
> > >>
> > >> Workarounds:
> > >> - Split buffers into smaller pieces in the kernel. It's not 
> > >> necessary to manage memory at page granularity (64KB). Splitting 
> > >> buffers into 16MB-large pieces might not be optimal but it would 
> > >> be a significant improvement.
> > >> - Or do the same in Mesa. This would prevent inter-process and 
> > >> inter-API buffer sharing for split buffers (DRI, OpenCL), but we 
> > >> would at least verify how much the situation improves.
> > >>
> > >> Other issues sharing the same cause:
> > >> - Allocations requesting 1/3 or more VRAM have a high chance of 
> > >> failing. It's generally not possible to allocate 1/2 or more VRAM 
> > >> as one buffer.
> > >>
> > >> Comments welcome,
> > >>
> > >> Marek
> > >> _______________________________________________
> > >> amd-gfx mailing list
> > >> amd-gfx at lists.freedesktop.org
> > >> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
> > >
> > >
> > > _______________________________________________
> > > amd-gfx mailing list
> > > amd-gfx at lists.freedesktop.org
> > > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
> >
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx at lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
> _______________________________________________
> amd-gfx mailing list
> amd-gfx at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx