[Intel-gfx] [PATCH 00/24] PPGTT dynamic page allocations

[Chris Wilson]

On Fri, Dec 19, 2014 at 02:29:57PM +0100, Daniel Vetter wrote:
> On Fri, Dec 19, 2014 at 01:10:40PM +0000, Chris Wilson wrote:
> > On Fri, Dec 19, 2014 at 11:13:51AM +0100, Daniel Vetter wrote:
> > > On Fri, Dec 19, 2014 at 08:50:09AM +0000, Chris Wilson wrote:
> > > > On Fri, Dec 19, 2014 at 09:37:52AM +0100, Daniel Vetter wrote:
> > > > > On Fri, Dec 19, 2014 at 08:31:03AM +0000, Chris Wilson wrote:
> > > > > > On Thu, Dec 18, 2014 at 10:16:22PM +0100, Daniel Vetter wrote:
> > > > > > > On Thu, Dec 18, 2014 at 05:09:57PM +0000, Michel Thierry wrote:
> > > > > > > > This new version tries to remove as many unnecessary changes as possible from
> > > > > > > > the previous RFC.
> > > > > > > >  
> > > > > > > > For GEN8, it has also been extended to work in logical ring submission (lrc)
> > > > > > > > mode, as it will be the preferred mode of operation.
> > > > > > > > I also tried to update the lrc code at the same time the ppgtt refactoring
> > > > > > > > occurred, leaving only one patch that is exclusively for lrc.
> > > > > > > > 
> > > > > > > > This list can be seen in 3 parts:
> > > > > > > > [01-10] Include code rework for PPGTT (all GENs).
> > > > > > > > [11-14] Adds page table allocation for GEN6/GEN7
> > > > > > > > [15-24] Enables dynamic allocation in GEN8. It is enabled for both legacy
> > > > > > > > and execlist submission modes.
> > > > > > > > 
> > > > > > > > Ben Widawsky (23):
> > > > > > > >   drm/i915: Add some extra guards in evict_vm
> > > > > > > >   drm/i915/trace: Fix offsets for 64b
> > > > > > > >   drm/i915: Rename to GEN8_LEGACY_PDPES
> > > > > > > >   drm/i915: Setup less PPGTT on failed pagedir
> > > > > > > >   drm/i915/gen8: Un-hardcode number of page directories
> > > > > > > >   drm/i915: Range clearing is PPGTT agnostic
> > > > > > > >   drm/i915: page table abstractions
> > > > > > > >   drm/i915: Complete page table structures
> > > > > > > >   drm/i915: Create page table allocators
> > > > > > > >   drm/i915: Track GEN6 page table usage
> > > > > > > >   drm/i915: Extract context switch skip logic
> > > > > > > >   drm/i915: Track page table reload need
> > > > > > > >   drm/i915: Initialize all contexts
> > > > > > > >   drm/i915: Finish gen6/7 dynamic page table allocation
> > > > > > > >   drm/i915/bdw: Use dynamic allocation idioms on free
> > > > > > > >   drm/i915/bdw: pagedirs rework allocation
> > > > > > > >   drm/i915/bdw: pagetable allocation rework
> > > > > > > >   drm/i915/bdw: Update pdp switch and point unused PDPs to scratch page
> > > > > > > >   drm/i915: num_pd_pages/num_pd_entries isn't useful
> > > > > > > >   drm/i915: Extract PPGTT param from pagedir alloc
> > > > > > > >   drm/i915/bdw: Split out mappings
> > > > > > > >   drm/i915/bdw: begin bitmap tracking
> > > > > > > >   drm/i915/bdw: Dynamic page table allocations
> > > > > > > > 
> > > > > > > > Michel Thierry (1):
> > > > > > > >   drm/i915/bdw: Dynamic page table allocations in lrc mode
> > > > > > > 
> > > > > > > Ok, I've tried to read through this series again and I definitely see a
> > > > > > > bit clearer, but it's still fairly confusing to me. I think that's just
> > > > > > > the long history of this patch series - often it seems to do something and
> > > > > > > then undo it again in some later patch. Which doesn't help understanding
> > > > > > > it.
> > > > > > > 
> > > > > > > I've replied with a few comments. Imo the way forward with this is to
> > > > > > > read, understand and review it from the beginning and merge while that's
> > > > > > > happening. It will probably take a few rounds until all the confusion is
> > > > > > > cleared up and we've reached the last patch.
> > > > > > 
> > > > > > I honestly think this is starting off in the wrong direction. The first
> > > > > > task, imo, is to make the current PD swappable. Then, we can introduce
> > > > > > infrastructure to do deferred page allocation, hopefully combining with
> > > > > > an approach to allow userspace to similarly defer their page allocation.
> > > > > 
> > > > > Thus far things started to blow up because we need a bit too much memory
> > > > > in testcases so we need this. We'll probably need the pd swapping
> > > > > eventually, too. But only on gen7, and atm it doesn't look good for ppgtt
> > > > > on gen7 still. Given that I think the priorities are right for now.
> > > > > 
> > > > > But yeah we need to keep this in mind for gen7 full ppgtt.
> > > > 
> > > > Certainly making the toplevel PD swappable on BDW is less of a priority,
> > > > since that should also be already done as part of execlists (and it is so
> > > > much smaller). I would like to see the gen6/7 deferred allocation
> > > > removed from this series, as I still consider it to be the wrong initial
> > > > approach (and making the current gen7 full-ppgtt swappable is rather
> > > > trivial).
> > > 
> > > I don't understand why you'd want to hold up the delayed pagetable alloc
> > > until the pd is evictable. Imo these are fairly orthogoanl issues: pd
> > > eviction tries to make ggtt space reclaimable, while deferred pagetable
> > > alloc ensure that we don't alloc 2M of pagetables (which are system
> > > memory) in the lower level when not needed.
> > 
> > There is only one level of PD on ivb, and it is 2M of pinned memory.
> 
> I think we have a mixup of nomeclatura here. The thing that takes 512x4k
> on ivb are imo the pagetables, whereas the page directory entries are the
> 512 entries in the global gtt table. We can't swap those entries out, but
> we could evict them from the ggtt.

Page Directory vs Page Directory Entries. Of course they can be written
to a swapfile when idle and repinned before use.

> > > So imo we can go ahead with
> > > both in parallel. There might be some minor conflicts between this series
> > > and making pd ggtt blocks evictable around pd reloading, but noting
> > > fundamental.
> > 
> > The trivial way to make it work, is to make the PD an obj. That then
> > ties into the existing vma/obj management and debug/error
> > infrastructure. Having deferred allocation for objects is also on the
> > wishlist.
> >  
> > > Or are we talking past each another? Just wondering since you call it
> > > "swappable pd" and I can't really think of a way we could swap out pds to
> > > disk on gen7 (they just block ggtt space which can't be used for real ptes
> > > any more).
> > 
> > Of course we can swap the inactive page directory tables.
> 
> See above, I don't think we can. Furthermore the vm doesn't bother with
> making page tables or directories reclaimable afaik (we don't need to swap
> them out, we can simply free them when everything in them is evicted). So
> I don't think that's worth the bother.
> 
> If we really have a problem with pagetables we can add it to the oom
> shrinker: Walking all vms and dropping all pagetables for completely empty
> vms would be conceptually really simple. And there shouldn't be anything
> left really when all objects are evicted for a ppgtt vm.

It is trivial to do so right now. If you look at the problem, you will
see that is a requirement to do things like execlists correctly, rather
than the current hodgepodge. In addition, we get a host of debugging
insight for free.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre