[PATCH hmm 02/15] mm/mmu_notifier: add an interval tree notifier

[Jason Gunthorpe]

On Mon, Oct 21, 2019 at 02:30:56PM -0400, Jerome Glisse wrote:

> > +/**
> > + * mmu_range_read_retry - End a read side critical section against a VA range
> > + * mrn: The range under lock
> > + * seq: The return of the paired mmu_range_read_begin()
> > + *
> > + * This MUST be called under a user provided lock that is also held
> > + * unconditionally by op->invalidate(). That lock provides the required SMP
> > + * barrier for handling invalidate_seq.
> > + *
> > + * Each call should be paired with a single mmu_range_read_begin() and
> > + * should be used to conclude the read side.
> > + *
> > + * Returns true if an invalidation collided with this critical section, and
> > + * the caller should retry.
> > + */
> > +static inline bool mmu_range_read_retry(struct mmu_range_notifier *mrn,
> > +					unsigned long seq)
> > +{
> > +	return READ_ONCE(mrn->invalidate_seq) != seq;
> > +}
> 
> What about calling this mmu_range_read_end() instead ? To match
> with the mmu_range_read_begin().

_end make some sense too, but I picked _retry for symmetry with the
seqcount_* family of functions which used retry.

I think retry makes it clearer that it is expected to fail and retry
is required.

> > +	/*
> > +	 * The inv_end incorporates a deferred mechanism like rtnl. Adds and
> 
> The rtnl reference is lost on people unfamiliar with the network :)
> code maybe like rtnl_lock()/rtnl_unlock() so people have a chance to
> grep the right function. Assuming i am myself getting the right
> reference :)

Yep, you got it, I will update

> > +	/*
> > +	 * mrn->invalidate_seq is always set to an odd value. This ensures
> > +	 * that if seq does wrap we will always clear the below sleep in some
> > +	 * reasonable time as mmn_mm->invalidate_seq is even in the idle
> > +	 * state.
> 
> I think this comment should be with the struct mmu_range_notifier
> definition and you should just point to it from here as the same
> comment would be useful down below.

I had it here because it is critical to understanding the wait_event
and why it doesn't just block indefinitely, but yes this property
comes up below too which refers back here.

Fundamentally this wait event is why this approach to keep an odd
value in the mrn is used.

> > -int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range)
> > +static int mn_itree_invalidate(struct mmu_notifier_mm *mmn_mm,
> > +				     const struct mmu_notifier_range *range)
> > +{
> > +	struct mmu_range_notifier *mrn;
> > +	unsigned long cur_seq;
> > +
> > +	for (mrn = mn_itree_inv_start_range(mmn_mm, range, &cur_seq); mrn;
> > +	     mrn = mn_itree_inv_next(mrn, range)) {
> > +		bool ret;
> > +
> > +		WRITE_ONCE(mrn->invalidate_seq, cur_seq);
> > +		ret = mrn->ops->invalidate(mrn, range);
> > +		if (!ret && !WARN_ON(mmu_notifier_range_blockable(range)))
> 
> Isn't the logic wrong here ? We want to warn if the range
> was mark as blockable and invalidate returned false. Also
> we went to backoff no matter what if the invalidate return
> false ie:

If invalidate returned false and the caller is blockable then we do
not want to return, we must continue processing other ranges - to try
to cope with the defective driver.

Callers in blocking mode ignore the return value and go ahead to
invalidate..

Would it be clearer as 

if (!ret) {
   if (WARN_ON(mmu_notifier_range_blockable(range)))
       continue;
   goto out_would_block;
}

?

> > @@ -284,21 +589,22 @@ int __mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm)
> >  		 * the write side of the mmap_sem.
> >  		 */
> >  		mmu_notifier_mm =
> > -			kmalloc(sizeof(struct mmu_notifier_mm), GFP_KERNEL);
> > +			kzalloc(sizeof(struct mmu_notifier_mm), GFP_KERNEL);
> >  		if (!mmu_notifier_mm)
> >  			return -ENOMEM;
> >  
> >  		INIT_HLIST_HEAD(&mmu_notifier_mm->list);
> >  		spin_lock_init(&mmu_notifier_mm->lock);
> > +		mmu_notifier_mm->invalidate_seq = 2;
> 
> Why starting at 2 ?

Good question. If everything is coded properly the starting value
doesn't matter

I left it like this because it makes debugging a tiny bit simpler, ie
if you print the seq number then the first mmu_range_notififers will
get 1 as their intial seq (see __mmu_range_notifier_insert) instead of
ULONG_MAX

> > +		mmu_notifier_mm->itree = RB_ROOT_CACHED;
> > +		init_waitqueue_head(&mmu_notifier_mm->wq);
> > +		INIT_HLIST_HEAD(&mmu_notifier_mm->deferred_list);
> >  	}
> >  
> >  	ret = mm_take_all_locks(mm);
> >  	if (unlikely(ret))
> >  		goto out_clean;
> >  
> > -	/* Pairs with the mmdrop in mmu_notifier_unregister_* */
> > -	mmgrab(mm);
> > -
> >  	/*
> >  	 * Serialize the update against mmu_notifier_unregister. A
> >  	 * side note: mmu_notifier_release can't run concurrently with
> > @@ -306,13 +612,26 @@ int __mmu_notifier_register(struct mmu_notifier *mn, struct mm_struct *mm)
> >  	 * current->mm or explicitly with get_task_mm() or similar).
> >  	 * We can't race against any other mmu notifier method either
> >  	 * thanks to mm_take_all_locks().
> > +	 *
> > +	 * release semantics are provided for users not inside a lock covered
> > +	 * by mm_take_all_locks(). acquire can only be used while holding the
> > +	 * mmgrab or mmget, and is safe because once created the
> > +	 * mmu_notififer_mm is not freed until the mm is destroyed.
> >  	 */
> >  	if (mmu_notifier_mm)
> > -		mm->mmu_notifier_mm = mmu_notifier_mm;
> > +		smp_store_release(&mm->mmu_notifier_mm, mmu_notifier_mm);
> 
> I do not understand why you need the release semantics here, we
> are under the mmap_sem in write mode when we release it the lock
> barrier will make sure anyone else sees the new mmu_notifier_mm

It pairs with the smp_load_acquire() in mmu_range_notifier_insert()
which is not called with the mmap_sem held. 

Since that reader is not locked we need release semantics here to
ensure the unlocked reader sees a fully initinalized mmu_notifier_mm
structure when it observes the pointer.

> > +/**
> > + * mmu_range_notifier_insert - Insert a range notifier
> > + * @mrn: Range notifier to register
> > + * @start: Starting virtual address to monitor
> > + * @length: Length of the range to monitor
> > + * @mm : mm_struct to attach to
> > + *
> > + * This function subscribes the range notifier for notifications from the mm.
> > + * Upon return the ops related to mmu_range_notifier will be called whenever
> > + * an event that intersects with the given range occurs.
> > + *
> > + * Upon return the range_notifier may not be present in the interval tree yet.
> > + * The caller must use the normal range notifier locking flow via
> > + * mmu_range_read_begin() to establish SPTEs for this range.
> > + */
> > +int mmu_range_notifier_insert(struct mmu_range_notifier *mrn,
> > +			      unsigned long start, unsigned long length,
> > +			      struct mm_struct *mm)
> > +{
> > +	struct mmu_notifier_mm *mmn_mm;
> > +	int ret;
> > +
> > +	might_lock(&mm->mmap_sem);
> > +
> > +	mmn_mm = smp_load_acquire(&mm->mmu_notifier_mm);

Right here we don't have the mmap_sem so this load is unlocked.

If we observe !mmn_mm we must also observe all stores done to set it
up. Ie we have to observe the spin_lock_init, RB_ROOT_CACHED/etc

> > +	if (!mmn_mm || !mmn_mm->has_interval) {
> > +		ret = mmu_notifier_register(NULL, mm);
> > +		if (ret)
> > +			return ret;
> > +		mmn_mm = mm->mmu_notifier_mm;
> > +	}
> > +	return __mmu_range_notifier_insert(mrn, start, length, mmn_mm, mm);
> > +}
> > +EXPORT_SYMBOL_GPL(mmu_range_notifier_insert);
> > +
> > +int mmu_range_notifier_insert_locked(struct mmu_range_notifier *mrn,
> > +				     unsigned long start, unsigned long length,
> > +				     struct mm_struct *mm)
> > +{
> > +	struct mmu_notifier_mm *mmn_mm;
> > +	int ret;
> > +
> > +	lockdep_assert_held_write(&mm->mmap_sem);
> > +
> > +	mmn_mm = mm->mmu_notifier_mm;
> 
> Shouldn't you be using smp_load_acquire() ?

This function is called while holding the mmap_sem. As you noted above
all writers to mm->mmu_notifier_mm hold the write side of mmap_sem,
thus here the read side is fully locked and doesn't need the acquire.

Note the lockdep annotations marking the expected locking enviroment
for the two functions.

Thanks,
Jason