[PATCH v6 12/12] drm/xe: Increase the XE_PL_TT watermark

[Matthew Brost]

On Mon, Aug 05, 2024 at 12:35:34PM -0600, Souza, Jose wrote:
> On Wed, 2024-07-03 at 17:38 +0200, Thomas Hellström wrote:
> > The XE_PL_TT watermark was set to 50% of system memory.
> > The idea behind that was unclear since the net effect is that
> > TT memory will be evicted to TTM_PL_SYSTEM memory if that
> > watermark is exceeded, requiring PPGTT rebinds and dma
> > remapping. But there is no similar watermark for TTM_PL_SYSTEM
> > memory.
> > 
> > The TTM functionality that tries to swap out system memory to
> > shmem objects if a 50% limit of total system memory is reached
> > is orthogonal to this, and with the shrinker added, it's no
> > longer in effect.
> > 
> > Replace the 50% TTM_PL_TT limit with a 100% limit, in effect
> > allowing all graphics memory to be bound to the device unless it
> > has been swapped out by the shrinker.
> 
> Sorry if I missed some patch changing it but I did not found in this series anything changing the 50% limit in ttm_global_init().
> When I debugged some Vulkan tests allocate a lot of memory, the reason that KMD was not allocating memory wash this ttm_global limit that is shared
> with all devices using TTM.
> 

I'm reviewing this series and starting make sense of all this.

Thomas please correct me if I'm wrong here...

The limit set in ttm_global_init is the watermark for the TTM pool where
if exceeded upon freeing a BO's pages the pages are actually freed
rather than just returning to the TTM pool cache. The global watermark
is reason why in issue #2438 it observed a bunch of memory is still
consumed when nothing is running or any BOs exist - pages are being
cached in the TTM pool. The global watermark doesn't actually limit the
amount system memory TTM can allocate. A shrinker also exists which can
free cached pages in the TTM pool if memory pressure exists or 'echo 3 >
/proc/sys/vm/drop_caches' is done.

The watermark changed in this patch, is the actual limit for the number
of pages we can allocate for BOs. With a shrinker hooked into BOs, we
now can freely allocate all of the system pages for BOs and if memory
pressure exists idle BOs pages are swapped to shmem via the shrinker and
restored upon next GPU use.

Matt

> > 
> > Signed-off-by: Thomas Hellström <thomas.hellstrom at linux.intel.com>
> > ---
> >  drivers/gpu/drm/xe/xe_ttm_sys_mgr.c | 3 +--
> >  1 file changed, 1 insertion(+), 2 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c b/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c
> > index 9844a8edbfe1..d38b91872da3 100644
> > --- a/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c
> > +++ b/drivers/gpu/drm/xe/xe_ttm_sys_mgr.c
> > @@ -108,9 +108,8 @@ int xe_ttm_sys_mgr_init(struct xe_device *xe)
> >  	u64 gtt_size;
> >  
> >  	si_meminfo(&si);
> > +	/* Potentially restrict amount of TT memory here. */
> >  	gtt_size = (u64)si.totalram * si.mem_unit;
> > -	/* TTM limits allocation of all TTM devices by 50% of system memory */
> > -	gtt_size /= 2;
> >  
> >  	man->use_tt = true;
> >  	man->func = &xe_ttm_sys_mgr_func;
>