[Intel-gfx] [PATCH 00/24] PPGTT dynamic page allocations

[Daniel Vetter]

On Fri, Dec 19, 2014 at 01:10:40PM +0000, Chris Wilson wrote:
> On Fri, Dec 19, 2014 at 11:13:51AM +0100, Daniel Vetter wrote:
> > On Fri, Dec 19, 2014 at 08:50:09AM +0000, Chris Wilson wrote:
> > > On Fri, Dec 19, 2014 at 09:37:52AM +0100, Daniel Vetter wrote:
> > > > On Fri, Dec 19, 2014 at 08:31:03AM +0000, Chris Wilson wrote:
> > > > > On Thu, Dec 18, 2014 at 10:16:22PM +0100, Daniel Vetter wrote:
> > > > > > On Thu, Dec 18, 2014 at 05:09:57PM +0000, Michel Thierry wrote:
> > > > > > > This new version tries to remove as many unnecessary changes as possible from
> > > > > > > the previous RFC.
> > > > > > >  
> > > > > > > For GEN8, it has also been extended to work in logical ring submission (lrc)
> > > > > > > mode, as it will be the preferred mode of operation.
> > > > > > > I also tried to update the lrc code at the same time the ppgtt refactoring
> > > > > > > occurred, leaving only one patch that is exclusively for lrc.
> > > > > > > 
> > > > > > > This list can be seen in 3 parts:
> > > > > > > [01-10] Include code rework for PPGTT (all GENs).
> > > > > > > [11-14] Adds page table allocation for GEN6/GEN7
> > > > > > > [15-24] Enables dynamic allocation in GEN8. It is enabled for both legacy
> > > > > > > and execlist submission modes.
> > > > > > > 
> > > > > > > Ben Widawsky (23):
> > > > > > >   drm/i915: Add some extra guards in evict_vm
> > > > > > >   drm/i915/trace: Fix offsets for 64b
> > > > > > >   drm/i915: Rename to GEN8_LEGACY_PDPES
> > > > > > >   drm/i915: Setup less PPGTT on failed pagedir
> > > > > > >   drm/i915/gen8: Un-hardcode number of page directories
> > > > > > >   drm/i915: Range clearing is PPGTT agnostic
> > > > > > >   drm/i915: page table abstractions
> > > > > > >   drm/i915: Complete page table structures
> > > > > > >   drm/i915: Create page table allocators
> > > > > > >   drm/i915: Track GEN6 page table usage
> > > > > > >   drm/i915: Extract context switch skip logic
> > > > > > >   drm/i915: Track page table reload need
> > > > > > >   drm/i915: Initialize all contexts
> > > > > > >   drm/i915: Finish gen6/7 dynamic page table allocation
> > > > > > >   drm/i915/bdw: Use dynamic allocation idioms on free
> > > > > > >   drm/i915/bdw: pagedirs rework allocation
> > > > > > >   drm/i915/bdw: pagetable allocation rework
> > > > > > >   drm/i915/bdw: Update pdp switch and point unused PDPs to scratch page
> > > > > > >   drm/i915: num_pd_pages/num_pd_entries isn't useful
> > > > > > >   drm/i915: Extract PPGTT param from pagedir alloc
> > > > > > >   drm/i915/bdw: Split out mappings
> > > > > > >   drm/i915/bdw: begin bitmap tracking
> > > > > > >   drm/i915/bdw: Dynamic page table allocations
> > > > > > > 
> > > > > > > Michel Thierry (1):
> > > > > > >   drm/i915/bdw: Dynamic page table allocations in lrc mode
> > > > > > 
> > > > > > Ok, I've tried to read through this series again and I definitely see a
> > > > > > bit clearer, but it's still fairly confusing to me. I think that's just
> > > > > > the long history of this patch series - often it seems to do something and
> > > > > > then undo it again in some later patch. Which doesn't help understanding
> > > > > > it.
> > > > > > 
> > > > > > I've replied with a few comments. Imo the way forward with this is to
> > > > > > read, understand and review it from the beginning and merge while that's
> > > > > > happening. It will probably take a few rounds until all the confusion is
> > > > > > cleared up and we've reached the last patch.
> > > > > 
> > > > > I honestly think this is starting off in the wrong direction. The first
> > > > > task, imo, is to make the current PD swappable. Then, we can introduce
> > > > > infrastructure to do deferred page allocation, hopefully combining with
> > > > > an approach to allow userspace to similarly defer their page allocation.
> > > > 
> > > > Thus far things started to blow up because we need a bit too much memory
> > > > in testcases so we need this. We'll probably need the pd swapping
> > > > eventually, too. But only on gen7, and atm it doesn't look good for ppgtt
> > > > on gen7 still. Given that I think the priorities are right for now.
> > > > 
> > > > But yeah we need to keep this in mind for gen7 full ppgtt.
> > > 
> > > Certainly making the toplevel PD swappable on BDW is less of a priority,
> > > since that should also be already done as part of execlists (and it is so
> > > much smaller). I would like to see the gen6/7 deferred allocation
> > > removed from this series, as I still consider it to be the wrong initial
> > > approach (and making the current gen7 full-ppgtt swappable is rather
> > > trivial).
> > 
> > I don't understand why you'd want to hold up the delayed pagetable alloc
> > until the pd is evictable. Imo these are fairly orthogoanl issues: pd
> > eviction tries to make ggtt space reclaimable, while deferred pagetable
> > alloc ensure that we don't alloc 2M of pagetables (which are system
> > memory) in the lower level when not needed.
> 
> There is only one level of PD on ivb, and it is 2M of pinned memory.

I think we have a mixup of nomeclatura here. The thing that takes 512x4k
on ivb are imo the pagetables, whereas the page directory entries are the
512 entries in the global gtt table. We can't swap those entries out, but
we could evict them from the ggtt.

> > So imo we can go ahead with
> > both in parallel. There might be some minor conflicts between this series
> > and making pd ggtt blocks evictable around pd reloading, but noting
> > fundamental.
> 
> The trivial way to make it work, is to make the PD an obj. That then
> ties into the existing vma/obj management and debug/error
> infrastructure. Having deferred allocation for objects is also on the
> wishlist.
>  
> > Or are we talking past each another? Just wondering since you call it
> > "swappable pd" and I can't really think of a way we could swap out pds to
> > disk on gen7 (they just block ggtt space which can't be used for real ptes
> > any more).
> 
> Of course we can swap the inactive page directory tables.

See above, I don't think we can. Furthermore the vm doesn't bother with
making page tables or directories reclaimable afaik (we don't need to swap
them out, we can simply free them when everything in them is evicted). So
I don't think that's worth the bother.

If we really have a problem with pagetables we can add it to the oom
shrinker: Walking all vms and dropping all pagetables for completely empty
vms would be conceptually really simple. And there shouldn't be anything
left really when all objects are evicted for a ppgtt vm.
-DAniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch