<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">On 2024-04-23 18:15, Felix Kuehling
wrote:<br>
</div>
<blockquote type="cite" cite="mid:21046a73-459f-4dc0-90b3-adef3ef6319b@amd.com">On
2024-04-23 11:28, Philip Yang wrote:
<br>
<blockquote type="cite">If the BO pages pinned for RDMA is not
contiguous on VRAM, evict it to
<br>
system memory first to free the VRAM space, then allocate
contiguous
<br>
VRAM space, and then move it from system memory back to VRAM.
<br>
<br>
Signed-off-by: Philip Yang <a class="moz-txt-link-rfc2396E" href="mailto:Philip.Yang@amd.com"><Philip.Yang@amd.com></a>
<br>
---
<br>
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 16
+++++++++++++++-
<br>
1 file changed, 15 insertions(+), 1 deletion(-)
<br>
<br>
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
<br>
index ef9154043757..5d118e5580ce 100644
<br>
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
<br>
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
<br>
@@ -1470,13 +1470,27 @@ static int
amdgpu_amdkfd_gpuvm_pin_bo(struct amdgpu_bo *bo, u32 domain)
<br>
if (unlikely(ret))
<br>
return ret;
<br>
+ if (bo->flags & AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS)
{
<br>
+ /*
<br>
+ * If bo is not contiguous on VRAM, move to system
memory first to ensure
<br>
+ * we can get contiguous VRAM space after evicting
other BOs.
<br>
+ */
<br>
+ if (!(bo->tbo.resource->placement &
TTM_PL_FLAG_CONTIGUOUS)) {
<br>
+ ret = amdgpu_amdkfd_bo_validate(bo,
AMDGPU_GEM_DOMAIN_GTT, false);
<br>
</blockquote>
<br>
amdgpu_amdkfd_bo_validate is meant for use in kernel threads. It
always runs uninterruptible. I believe pin_bo runs in the context
of ioctls from user mode. So it should be interruptible.
<br>
</blockquote>
<p>yes, pin_bo is in the context of user mode, from KFD alloc memory
or from rdma driver get pages, should use interruptible wait.</p>
<p>amdgpu_amdkfd_bo_validate is currently used by kernel threads and
ioctl amdgpu_amdkfd_add_gws_to_process (this seems bug), does it
make sense to add parameter interruptible, then we can remove many
duplicate code amdgpu_bo_placement_from_domain + ttm_bo_validate
or I can fix it here and leave the cleanup and bug fix in the
future?</p>
<p>Regards,</p>
<p>Philip<br>
</p>
<blockquote type="cite" cite="mid:21046a73-459f-4dc0-90b3-adef3ef6319b@amd.com">
<br>
Regards,
<br>
Felix
<br>
<br>
<br>
<blockquote type="cite">+ if (unlikely(ret)) {
<br>
+ pr_debug("validate bo 0x%p to GTT failed %d\n",
&bo->tbo, ret);
<br>
+ goto out;
<br>
+ }
<br>
+ }
<br>
+ }
<br>
+
<br>
ret = amdgpu_bo_pin_restricted(bo, domain, 0, 0);
<br>
if (ret)
<br>
pr_err("Error in Pinning BO to domain: %d\n", domain);
<br>
amdgpu_bo_sync_wait(bo, AMDGPU_FENCE_OWNER_KFD, false);
<br>
+out:
<br>
amdgpu_bo_unreserve(bo);
<br>
-
<br>
return ret;
<br>
}
<br>
</blockquote>
</blockquote>
</body>
</html>