[Intel-gfx] [RFC PATCH] drm/ttm: Fix swapping dereferences of freed memory

[Thomas Hellström]

On Fri, 2021-05-28 at 09:16 +0200, Christian König wrote:
> Am 27.05.21 um 17:51 schrieb Thomas Hellström:
> > On Thu, 2021-05-27 at 17:32 +0200, Christian König wrote:
> > > Am 27.05.21 um 17:05 schrieb Thomas Hellström:
> > > > On Thu, 2021-05-27 at 17:01 +0200, Thomas Hellström wrote:
> > > > > On Thu, 2021-05-27 at 16:54 +0200, Christian König wrote:
> > > > > > Am 27.05.21 um 16:19 schrieb Thomas Hellström:
> > > > > > > The swapping code was dereference bo->ttm pointers
> > > > > > > without
> > > > > > > having
> > > > > > > the
> > > > > > > dma-resv lock held. Also it might try to swap out
> > > > > > > unpopulated
> > > > > > > bos.
> > > > > > > 
> > > > > > > Fix this by moving the bo->ttm dereference until we have
> > > > > > > the
> > > > > > > reservation
> > > > > > > lock. Check that the ttm_tt is populated after the
> > > > > > > swap_notify
> > > > > > > callback.
> > > > > > > 
> > > > > > > Signed-off-by: Thomas Hellström
> > > > > > > <thomas.hellstrom at linux.intel.com>
> > > > > > > ---
> > > > > > >     drivers/gpu/drm/ttm/ttm_bo.c     | 16
> > > > > > > +++++++++++++++-
> > > > > > >     drivers/gpu/drm/ttm/ttm_device.c |  8 +++-----
> > > > > > >     2 files changed, 18 insertions(+), 6 deletions(-)
> > > > > > > 
> > > > > > > diff --git a/drivers/gpu/drm/ttm/ttm_bo.c
> > > > > > > b/drivers/gpu/drm/ttm/ttm_bo.c
> > > > > > > index 9f53506a82fc..86213d37657b 100644
> > > > > > > --- a/drivers/gpu/drm/ttm/ttm_bo.c
> > > > > > > +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> > > > > > > @@ -1163,6 +1163,16 @@ int ttm_bo_swapout(struct
> > > > > > > ttm_buffer_object
> > > > > > > *bo, struct ttm_operation_ctx *ctx,
> > > > > > >           if (!ttm_bo_evict_swapout_allowable(bo, ctx,
> > > > > > > &place,
> > > > > > > &locked, NULL))
> > > > > > >                   return -EBUSY;
> > > > > > >     
> > > > > > > +       dma_resv_assert_held(bo->base.resv);
> > > > > > > +
> > > > > > > +       if (!bo->ttm ||
> > > > > > > +           bo->ttm->page_flags & TTM_PAGE_FLAG_SG ||
> > > > > > > +           bo->ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)
> > > > > > > {
> > > > > > > +               if (locked)
> > > > > > > +                       dma_resv_unlock(bo->base.resv);
> > > > > > > +               return -EBUSY;
> > > > > > > +       }
> > > > > > > +
> > > > > > >           if (!ttm_bo_get_unless_zero(bo)) {
> > > > > > >                   if (locked)
> > > > > > >                           dma_resv_unlock(bo->base.resv);
> > > > > > > @@ -1215,7 +1225,8 @@ int ttm_bo_swapout(struct
> > > > > > > ttm_buffer_object
> > > > > > > *bo, struct ttm_operation_ctx *ctx,
> > > > > > >           if (bo->bdev->funcs->swap_notify)
> > > > > > >                   bo->bdev->funcs->swap_notify(bo);
> > > > > > >     
> > > > > > > -       ret = ttm_tt_swapout(bo->bdev, bo->ttm,
> > > > > > > gfp_flags);
> > > > > > > +       if (ttm_tt_is_populated(bo->ttm))
> > > > > > > +               ret = ttm_tt_swapout(bo->bdev, bo->ttm,
> > > > > > > gfp_flags);
> > > > > > Exactly that is what I won't recommend. We would try to
> > > > > > swap
> > > > > > out
> > > > > > the
> > > > > > same BO over and over again with that.
> > > > > But we wouldn't since the BO is taken off the LRU and never
> > > > > re-
> > > > > added,
> > > > > 
> > > > > 
> > > > In fact, we'd probably might want to take the !bo->ttm bos off
> > > > the
> > > > LRU
> > > > as well..
> > > No, we don't want to take any BOs of the LRU unless they are
> > > pinned.
> > > 
> > > Adding a TT object or populating it doesn't necessarily put the
> > > BO
> > > back
> > > to the LRU.
> > OK, but swapped bos are also taken off the LRU list so these
> > unpopulated bos are just taking the same path. Only difference to
> > swapped is that they don't get read back on re-populate, but
> > typically
> > cleared.
> > 
> > But what would be the point of keeping swapped-out bos on the LRU
> > list?, particularly when we're iterating under a spinlock?
> > Shouldn't we try to re-add to LRU (if not already on an LRU) just
> > before populating? There aren't really that many calls in core TTM.
> 
> I want to avoid removing BOs from the LRU as much as possible since
> we 
> forgot on multiple places that we want to re-add them.
> 
> Conceptual I think the swapped BOs should have a separate memory
> domain, 
> this way we can ignore them cleanly when swapping things out.

Yes, that would of course work as well. Keeping them on the system LRU
is IMO highly undesirable.

> 
> Going to pick this patch up, modifying it a bit more and then pushing
> it 
> to drm-misc-fixes for upstreaming.

OK, I dropped the TTM fix for the purge-in-swap-notify from the i915
series, hoping that the reworked variant of this patch lands first.

Thanks,
Thomas

> 
> Thanks,
> Christian.
> 
> > 
> > /Thomas
> > 
> > 
> > 
> > 
> > 
> > > Christian.
> > > 
> > > > /Thomas
> > > > 
> > 
>