[PATCH rdma-core v3 4/6] pyverbs: Add dma-buf based MR support

[Daniel Vetter]

On Mon, Nov 30, 2020 at 12:36:42PM -0400, Jason Gunthorpe wrote:
> On Mon, Nov 30, 2020 at 05:04:43PM +0100, Daniel Vetter wrote:
> > On Mon, Nov 30, 2020 at 11:55:44AM -0400, Jason Gunthorpe wrote:
> > > On Mon, Nov 30, 2020 at 03:57:41PM +0100, Daniel Vetter wrote:
> > > > > +	err = ioctl(dri->fd, DRM_IOCTL_AMDGPU_GEM_CREATE, &gem_create);
> > > > > +	if (err)
> > > > > +		return err;
> > > > > +
> > > > > +	*handle = gem_create.out.handle;
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > > +static int radeon_alloc(struct dri *dri, size_t size, uint32_t *handle)
> > > > 
> > > > Tbh radeon chips are old enough I wouldn't care. Also doesn't support p2p
> > > > dma-buf, so always going to be in system memory when you share. Plus you
> > > > also need some more flags like I suggested above I think.
> > > 
> > > What about nouveau?
> > 
> > Reallistically chances that someone wants to use rdma together with the
> > upstream nouveau driver are roughly nil. Imo also needs someone with the
> > right hardware to make sure it works (since the flags are all kinda arcane
> > driver specific stuff testing is really needed).
> 
> Well, it would be helpful if we can test the mlx5 part of the
> implementation, and I have a lab stocked with nouveau compatible HW..
> 
> But you are right someone needs to test/etc, so this does not seem
> like Jianxin should worry

Ah yes sounds good. I can help with trying to find how to allocate vram
with nouveau if you don't find it. Caveat is that nouveau doesn't do
dynamic dma-buf exports and hence none of the intersting flows and also
not p2p. Not sure how much work it would be to roll that out (iirc it
wasnt that much amdgpu code really, just endless discussions on the
interface semantics and how to roll it out without breaking any of the
existing dma-buf users).

Another thing that just crossed my mind: Do we have a testcase for forcing
the eviction? Should be fairly easy to provoke with something like this:

- register vram-only buffer with mlx5 and do something that binds it
- allocate enough vram-only buffers to overfill vram (again figuring out
  how much vram you have is driver specific)
- touch each buffer with mmap. that should force the mlx5 buffer out. it
  might be that eviction isn't lru but preferentially idle buffers (i.e.
  not used by hw, so anything register to mlx5 won't qualify as first
  victims). so we might need to instead register a ton of buffers with
  mlx5 and access them through ibverbs
- do something with mlx5 again to force the rebinding and test it all
  keeps working

That entire invalidate/buffer move flow is the most complex interaction I
think.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch