[Intel-gfx] [PATCH 2/2] dma-buf/dma-fence: Add quick tests before dma_fence_remove_callback

[Daniel Vetter]

On Wed, Jul 15, 2020 at 2:40 PM Chris Wilson <chris at chris-wilson.co.uk> wrote:
> Quoting Chris Wilson (2020-07-15 13:21:43)
> > Quoting Daniel Vetter (2020-07-15 13:10:22)
> > > On Wed, Jul 15, 2020 at 11:49:05AM +0100, Chris Wilson wrote:
> > > > When waiting with a callback on the stack, we must remove the callback
> > > > upon wait completion. Since this will be notified by the fence signal
> > > > callback, the removal often contends with the fence->lock being held by
> > > > the signaler. We can look at the list entry to see if the callback was
> > > > already signaled before we take the contended lock.
> > > >
> > > > Signed-off-by: Chris Wilson <chris at chris-wilson.co.uk>
> > > > ---
> > > >  drivers/dma-buf/dma-fence.c | 3 +++
> > > >  1 file changed, 3 insertions(+)
> > > >
> > > > diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c
> > > > index 8d5bdfce638e..b910d7bc0854 100644
> > > > --- a/drivers/dma-buf/dma-fence.c
> > > > +++ b/drivers/dma-buf/dma-fence.c
> > > > @@ -420,6 +420,9 @@ dma_fence_remove_callback(struct dma_fence *fence, struct dma_fence_cb *cb)
> > > >       unsigned long flags;
> > > >       bool ret;
> > > >
> > > > +     if (list_empty(&cb->node))
> > >
> > > I was about to say "but the races" but then noticed that Paul fixed
> > > list_empty to use READ_ONCE like 5 years ago :-)
> >
> > I'm always going "when exactly do we need list_empty_careful()"?
> >
> > We can rule out a concurrent dma_fence_add_callback() for the same
> > dma_fence_cb, as that is a lost cause. So we only have to worry about
> > the concurrent list_del_init() from dma_fence_signal_locked(). So it's
> > the timing of
> >         list_del_init(): WRITE_ONCE(list->next, list)
> > vs
> >         READ_ONCE(list->next) == list
> > and we don't need to care about the trailing instructions in
> > list_del_init()...
> >
> > Wait that trailing instruction is actually important here if the
> > dma_fence_cb is on the stack, or other imminent free.
> >
> > Ok, this does need to be list_empty_careful!

Hm, tbh I'm not really clear what list_empty_careful does on top.
Seems to lack READ_ONCE, so either some really big trickery with
dependencies is going on, or I'm not even sure how this works without
locks.

I've now stared at list_empty_careful and a bunch of users quite a
bit, and I have now idea when you'd want to use it. Lockless you want
a READ_ONCE I think and a simple check, so list_empty. And just accept
that any time you race you'll go into the locked slowpath for "list
isn't empty". Also only works if the list_empty case is the "nothing
to do, job already done" case, since the other one just isn't
guaranteed to be accurate.

list_empty_careful just wraps a bunch more magic around that will make
this both worse, and more racy (if the compiler feels creative)
because no READ_ONCE or anything like that. I don't get what that
thing is for ...

> There's a further problem in that we call INIT_LIST_HEAD on the
> dma_fence_cb before the signal callback. So even if list_empty_careful()
> confirms the dma_fence_cb to be completely decoupled, the containing
> struct may still be inuse.

The kerneldoc of dma_fence_remove_callback() already has a very stern
warning that this will blow up if you don't hold a full reference or
otherwise control the lifetime of this stuff. So I don't think we have
to worry about any of that. Or do you mean something else?
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch