[RFC PATCH] drm/panfrost: Add support for mapping BOs on GPU page faults

[Rob Herring]

On Sun, Jun 16, 2019 at 11:15 PM Tomeu Vizoso
<tomeu.vizoso at collabora.com> wrote:
>
> On Fri, 14 Jun 2019 at 23:22, Rob Herring <robh at kernel.org> wrote:
> >
> > On Wed, Jun 12, 2019 at 6:55 AM Tomeu Vizoso <tomeu at tomeuvizoso.net> wrote:
> > >
> > > On Mon, 10 Jun 2019 at 19:06, Rob Herring <robh at kernel.org> wrote:
> > > >
> > > > The midgard/bifrost GPUs need to allocate GPU memory which is allocated
> > > > on GPU page faults and not pinned in memory. The vendor driver calls
> > > > this functionality GROW_ON_GPF.
> > > >
> > > > This implementation assumes that BOs allocated with the
> > > > PANFROST_BO_NOMAP flag are never mmapped or exported. Both of those may
> > > > actually work, but I'm unsure if there's some interaction there. It
> > > > would cause the whole object to be pinned in memory which would defeat
> > > > the point of this.
> > > >
> > > > Issues/questions/thoughts:
> > > >
> > > > What's the difference between i_mapping and f_mapping?
> > > >
> > > > What kind of clean-up on close is needed? Based on vgem faults, there
> > > > doesn't seem to be any refcounting. Assume userspace is responsible for
> > > > not freeing the BO while a page fault can occur?
> > >
> > > Aren't we taking a reference on all BOs that a job relates to and
> > > unreferencing them once the job is done? I would think that that's
> > > enough, or am I missing something?
> >
> > No, I think we're fine.
> >
> > > > What about evictions? Need to call mapping_set_unevictable()? Maybe we
> > > > want these pages to be swappable, but then we need some notification to
> > > > unmap them.
> > >
> > > I'm not sure there's much point in swapping out pages with lifetimes
> > > of a few milliseconds.
> >
> > The lifetime is *forever* though. If we don't allow swapping, then the
> > heap is grow only until the FD is closed. IIRC, the maximum size is on
> > the order of 1GB. Seems like you'd want to shrink it with some
> > trigger.
>
> I thought that the lifetime of the *contents* of the heap was that of
> the job chain that wrote them? Otherwise, only the GPU would know what
> can be discarded.

Yes, that's probably true. To take that to the extreme, we could
allocate and free the heap BO on each job chain. But we don't do that
because of the overhead. So mapping and unmapping is a similar trade
off of frequency vs. overhead. The question is when do we allow pages
to be swapped out (as that is unhandled in the current patch)? Our
choices are:

- at any time. This is what the patch currently does as we don't
prevent eviction. Though we'd need some mechanism to be notified when
a page is evicted which is currently missing.
- when a job finishes. We'd have to iterate thru BO's and mark pages
evict-able on NOMAP BOs. Not sure where we do that in the driver.
- when the BO is freed. This is the easiest to implement...

Rob