[PATCH hmm 02/15] mm/mmu_notifier: add an interval tree notifier
Jerome Glisse
jglisse at redhat.com
Mon Oct 21 19:11:57 UTC 2019
On Mon, Oct 21, 2019 at 06:54:25PM +0000, Jason Gunthorpe wrote:
> On Mon, Oct 21, 2019 at 02:30:56PM -0400, Jerome Glisse wrote:
>
> > > +/**
> > > + * mmu_range_read_retry - End a read side critical section against a VA range
> > > + * mrn: The range under lock
> > > + * seq: The return of the paired mmu_range_read_begin()
> > > + *
> > > + * This MUST be called under a user provided lock that is also held
> > > + * unconditionally by op->invalidate(). That lock provides the required SMP
> > > + * barrier for handling invalidate_seq.
> > > + *
> > > + * Each call should be paired with a single mmu_range_read_begin() and
> > > + * should be used to conclude the read side.
> > > + *
> > > + * Returns true if an invalidation collided with this critical section, and
> > > + * the caller should retry.
> > > + */
> > > +static inline bool mmu_range_read_retry(struct mmu_range_notifier *mrn,
> > > + unsigned long seq)
> > > +{
> > > + return READ_ONCE(mrn->invalidate_seq) != seq;
> > > +}
> >
> > What about calling this mmu_range_read_end() instead ? To match
> > with the mmu_range_read_begin().
>
> _end make some sense too, but I picked _retry for symmetry with the
> seqcount_* family of functions which used retry.
>
> I think retry makes it clearer that it is expected to fail and retry
> is required.
Fair enough.
>
> > > + /*
> > > + * The inv_end incorporates a deferred mechanism like rtnl. Adds and
> >
> > The rtnl reference is lost on people unfamiliar with the network :)
> > code maybe like rtnl_lock()/rtnl_unlock() so people have a chance to
> > grep the right function. Assuming i am myself getting the right
> > reference :)
>
> Yep, you got it, I will update
>
> > > + /*
> > > + * mrn->invalidate_seq is always set to an odd value. This ensures
> > > + * that if seq does wrap we will always clear the below sleep in some
> > > + * reasonable time as mmn_mm->invalidate_seq is even in the idle
> > > + * state.
> >
> > I think this comment should be with the struct mmu_range_notifier
> > definition and you should just point to it from here as the same
> > comment would be useful down below.
>
> I had it here because it is critical to understanding the wait_event
> and why it doesn't just block indefinitely, but yes this property
> comes up below too which refers back here.
>
> Fundamentally this wait event is why this approach to keep an odd
> value in the mrn is used.
The comment is fine, it is just i read the patch out of order and
in insert function i was pondering on why it must be odd while the
explanation was here. It is more a taste thing, i prefer comments
about this to be part of the struct definition comments so that
multiple place can refer to the same struct definition it is more
resiliant to code change as struct definition is always easy to
find and thus reference to it can be sprinkle all over where it is
necessary.
>
> > > -int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range)
> > > +static int mn_itree_invalidate(struct mmu_notifier_mm *mmn_mm,
> > > + const struct mmu_notifier_range *range)
> > > +{
> > > + struct mmu_range_notifier *mrn;
> > > + unsigned long cur_seq;
> > > +
> > > + for (mrn = mn_itree_inv_start_range(mmn_mm, range, &cur_seq); mrn;
> > > + mrn = mn_itree_inv_next(mrn, range)) {
> > > + bool ret;
> > > +
> > > + WRITE_ONCE(mrn->invalidate_seq, cur_seq);
> > > + ret = mrn->ops->invalidate(mrn, range);
> > > + if (!ret && !WARN_ON(mmu_notifier_range_blockable(range)))
> >
> > Isn't the logic wrong here ? We want to warn if the range
> > was mark as blockable and invalidate returned false. Also
> > we went to backoff no matter what if the invalidate return
> > false ie:
>
> If invalidate returned false and the caller is blockable then we do
> not want to return, we must continue processing other ranges - to try
> to cope with the defective driver.
>
> Callers in blocking mode ignore the return value and go ahead to
> invalidate..
>
> Would it be clearer as
>
> if (!ret) {
> if (WARN_ON(mmu_notifier_range_blockable(range)))
> continue;
> goto out_would_block;
> }
>
> ?
Yes look clearer to me at least.
>
> > > @@ -284,21 +589,22 @@ int __mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm)
> > > * the write side of the mmap_sem.
> > > */
> > > mmu_notifier_mm =
> > > - kmalloc(sizeof(struct mmu_notifier_mm), GFP_KERNEL);
> > > + kzalloc(sizeof(struct mmu_notifier_mm), GFP_KERNEL);
> > > if (!mmu_notifier_mm)
> > > return -ENOMEM;
> > >
> > > INIT_HLIST_HEAD(&mmu_notifier_mm->list);
> > > spin_lock_init(&mmu_notifier_mm->lock);
> > > + mmu_notifier_mm->invalidate_seq = 2;
> >
> > Why starting at 2 ?
>
> Good question. If everything is coded properly the starting value
> doesn't matter
>
> I left it like this because it makes debugging a tiny bit simpler, ie
> if you print the seq number then the first mmu_range_notififers will
> get 1 as their intial seq (see __mmu_range_notifier_insert) instead of
> ULONG_MAX
Yeah make sense.
>
> > > + mmu_notifier_mm->itree = RB_ROOT_CACHED;
> > > + init_waitqueue_head(&mmu_notifier_mm->wq);
> > > + INIT_HLIST_HEAD(&mmu_notifier_mm->deferred_list);
> > > }
> > >
> > > ret = mm_take_all_locks(mm);
> > > if (unlikely(ret))
> > > goto out_clean;
> > >
> > > - /* Pairs with the mmdrop in mmu_notifier_unregister_* */
> > > - mmgrab(mm);
> > > -
> > > /*
> > > * Serialize the update against mmu_notifier_unregister. A
> > > * side note: mmu_notifier_release can't run concurrently with
> > > @@ -306,13 +612,26 @@ int __mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm)
> > > * current->mm or explicitly with get_task_mm() or similar).
> > > * We can't race against any other mmu notifier method either
> > > * thanks to mm_take_all_locks().
> > > + *
> > > + * release semantics are provided for users not inside a lock covered
> > > + * by mm_take_all_locks(). acquire can only be used while holding the
> > > + * mmgrab or mmget, and is safe because once created the
> > > + * mmu_notififer_mm is not freed until the mm is destroyed.
> > > */
> > > if (mmu_notifier_mm)
> > > - mm->mmu_notifier_mm = mmu_notifier_mm;
> > > + smp_store_release(&mm->mmu_notifier_mm, mmu_notifier_mm);
> >
> > I do not understand why you need the release semantics here, we
> > are under the mmap_sem in write mode when we release it the lock
> > barrier will make sure anyone else sees the new mmu_notifier_mm
>
> It pairs with the smp_load_acquire() in mmu_range_notifier_insert()
> which is not called with the mmap_sem held.
>
> Since that reader is not locked we need release semantics here to
> ensure the unlocked reader sees a fully initinalized mmu_notifier_mm
> structure when it observes the pointer.
I thought the mm_take_all_locks() would have had a barrier and thus
that you could not see mmu_notifier struct partialy initialized. But
having the acquire/release as safety net does not hurt. Maybe add a
comment about the struct initialization needing to be visible before
pointer is set.
>
> > > +/**
> > > + * mmu_range_notifier_insert - Insert a range notifier
> > > + * @mrn: Range notifier to register
> > > + * @start: Starting virtual address to monitor
> > > + * @length: Length of the range to monitor
> > > + * @mm : mm_struct to attach to
> > > + *
> > > + * This function subscribes the range notifier for notifications from the mm.
> > > + * Upon return the ops related to mmu_range_notifier will be called whenever
> > > + * an event that intersects with the given range occurs.
> > > + *
> > > + * Upon return the range_notifier may not be present in the interval tree yet.
> > > + * The caller must use the normal range notifier locking flow via
> > > + * mmu_range_read_begin() to establish SPTEs for this range.
> > > + */
> > > +int mmu_range_notifier_insert(struct mmu_range_notifier *mrn,
> > > + unsigned long start, unsigned long length,
> > > + struct mm_struct *mm)
> > > +{
> > > + struct mmu_notifier_mm *mmn_mm;
> > > + int ret;
> > > +
> > > + might_lock(&mm->mmap_sem);
> > > +
> > > + mmn_mm = smp_load_acquire(&mm->mmu_notifier_mm);
>
> Right here we don't have the mmap_sem so this load is unlocked.
>
> If we observe !mmn_mm we must also observe all stores done to set it
> up. Ie we have to observe the spin_lock_init, RB_ROOT_CACHED/etc
>
> > > + if (!mmn_mm || !mmn_mm->has_interval) {
> > > + ret = mmu_notifier_register(NULL, mm);
> > > + if (ret)
> > > + return ret;
> > > + mmn_mm = mm->mmu_notifier_mm;
> > > + }
> > > + return __mmu_range_notifier_insert(mrn, start, length, mmn_mm, mm);
> > > +}
> > > +EXPORT_SYMBOL_GPL(mmu_range_notifier_insert);
> > > +
> > > +int mmu_range_notifier_insert_locked(struct mmu_range_notifier *mrn,
> > > + unsigned long start, unsigned long length,
> > > + struct mm_struct *mm)
> > > +{
> > > + struct mmu_notifier_mm *mmn_mm;
> > > + int ret;
> > > +
> > > + lockdep_assert_held_write(&mm->mmap_sem);
> > > +
> > > + mmn_mm = mm->mmu_notifier_mm;
> >
> > Shouldn't you be using smp_load_acquire() ?
>
> This function is called while holding the mmap_sem. As you noted above
> all writers to mm->mmu_notifier_mm hold the write side of mmap_sem,
> thus here the read side is fully locked and doesn't need the acquire.
>
> Note the lockdep annotations marking the expected locking enviroment
> for the two functions.
Yes i thought you had the acquire/release for some other reason
than struct init. It is fine here to not use the load_acquire.
Cheers,
Jérôme
More information about the dri-devel
mailing list