<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word"> <div class="WordSection1"> [AMD Official Use Only - General]<o:p></o:p> <o:p> </o:p> This seems to have broken CRIU restore. After I revert this, I can get CRIU restore working.<o:p></o:p> <o:p> </o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of Russell, Kent Sent: Tuesday, August 23, 2022 8:07 AM To: Kuehling, Felix <Felix.Kuehling@amd.com>; amd-gfx@lists.freedesktop.org Subject: Re: [PATCH] drm/amdkfd: Allocate doorbells only when needed<o:p></o:p> </div> </div> <o:p> </o:p> <div> No one else seems to have any thoughts on it, so Reviewed-by: Kent Russell <<a href="mailto:kent.Russell@amd.com">kent.Russell@amd.com</a>><o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> Kent<o:p></o:p> </div> <o:p> </o:p> <div class="MsoNormal" align="center" style="text-align:center"> <hr size="1" width="98%" align="center"> </div> <div id="divRplyFwdMsg"> From: amd-gfx <<a href="mailto:amd-gfx-bounces@lists.freedesktop.org">amd-gfx-bounces@lists.freedesktop.org</a>> on behalf of Russell, Kent <<a href="mailto:Kent.Russell@amd.com">Kent.Russell@amd.com</a>> Sent: Monday, August 22, 2022 10:10 AM To: Kuehling, Felix <<a href="mailto:Felix.Kuehling@amd.com">Felix.Kuehling@amd.com</a>>; <a href="mailto:amd-gfx@lists.freedesktop.org">amd-gfx@lists.freedesktop.org</a> <<a href="mailto:amd-gfx@lists.freedesktop.org">amd-gfx@lists.freedesktop.org</a>> Subject: RE: [PATCH] drm/amdkfd: Allocate doorbells only when needed <o:p></o:p> <div> <o:p></o:p> </div> </div> <div> <div> [AMD Official Use Only - General] I can throw an Acked-by: Kent Russell <<a href="mailto:kent.russell@amd.com">kent.russell@amd.com</a>> since we don't have an RB yet. Kent > -----Original Message----- > From: amd-gfx <<a href="mailto:amd-gfx-bounces@lists.freedesktop.org">amd-gfx-bounces@lists.freedesktop.org</a>> On Behalf Of Felix > Kuehling > Sent: Wednesday, August 3, 2022 2:56 PM > To: <a href="mailto:amd-gfx@lists.freedesktop.org">amd-gfx@lists.freedesktop.org</a> > Subject: [PATCH] drm/amdkfd: Allocate doorbells only when needed > > Only allocate doorbells when the first queue is created on a GPU or the > doorbells need to be mapped into CPU or GPU virtual address space. This > avoids allocating doorbells unnecessarily and can allow more processes > to use KFD on multi-GPU systems. > > Signed-off-by: Felix Kuehling <<a href="mailto:Felix.Kuehling@amd.com">Felix.Kuehling@amd.com</a>> > --- > drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 13 +++++++++++++ > drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c | 9 +++++++++ > drivers/gpu/drm/amd/amdkfd/kfd_process.c | 5 ----- > 3 files changed, 22 insertions(+), 5 deletions(-) > > diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c > b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c > index 2b3d8bc8f0aa..907f4711abce 100644 > --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c > +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c > @@ -327,6 +327,12 @@ static int kfd_ioctl_create_queue(struct file *filep, > struct kfd_process *p, > goto err_bind_process; > } > > + if (!pdd->doorbell_index && > + kfd_alloc_process_doorbells(dev, &pdd->doorbell_index) < 0) { > + err = -ENOMEM; > + goto err_alloc_doorbells; > + } > + > /* Starting with GFX11, wptr BOs must be mapped to GART for MES to > determine work > * on unmapped queues for usermode queue oversubscription (no > aggregated doorbell) > */ > @@ -404,6 +410,7 @@ static int kfd_ioctl_create_queue(struct file *filep, struct > kfd_process *p, > if (wptr_bo) > amdgpu_amdkfd_free_gtt_mem(dev->adev, wptr_bo); > err_wptr_map_gart: > +err_alloc_doorbells: > err_bind_process: > err_pdd: > mutex_unlock(&p->mutex); > @@ -1092,6 +1099,10 @@ static int kfd_ioctl_alloc_memory_of_gpu(struct file > *filep, > goto err_unlock; > } > offset = kfd_get_process_doorbells(pdd); > + if (!offset) { > + err = -ENOMEM; > + goto err_unlock; > + } > } else if (flags & KFD_IOC_ALLOC_MEM_FLAGS_MMIO_REMAP) { > if (args->size != PAGE_SIZE) { > err = -EINVAL; > @@ -2173,6 +2184,8 @@ static int criu_restore_memory_of_gpu(struct > kfd_process_device *pdd, > return -EINVAL; > > offset = kfd_get_process_doorbells(pdd); > + if (!offset) > + return -ENOMEM;<o:p></o:p> <o:p> </o:p> Looks like this is causing CRIU restore failure.<o:p></o:p> > } else if (bo_bucket->alloc_flags & > KFD_IOC_ALLOC_MEM_FLAGS_MMIO_REMAP) { > /* MMIO BOs need remapped bus address */ > if (bo_bucket->size != PAGE_SIZE) { > diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c > b/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c > index cb3d2ccc5100..b33798f89ef0 100644 > --- a/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c > +++ b/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c > @@ -157,6 +157,8 @@ int kfd_doorbell_mmap(struct kfd_dev *dev, struct > kfd_process *process, > > /* Calculate physical address of doorbell */ > address = kfd_get_process_doorbells(pdd); > + if (!address) > + return -ENOMEM; > vma->vm_flags |= VM_IO | VM_DONTCOPY | VM_DONTEXPAND | > VM_NORESERVE | > VM_DONTDUMP | VM_PFNMAP; > > @@ -275,6 +277,13 @@ uint64_t kfd_get_number_elems(struct kfd_dev *kfd) > > phys_addr_t kfd_get_process_doorbells(struct kfd_process_device *pdd) > { > + if (!pdd->doorbell_index) { > + int r = kfd_alloc_process_doorbells(pdd->dev, > + &pdd->doorbell_index); > + if (r) > + return 0; > + } > + > return pdd->dev->doorbell_base + > pdd->doorbell_index * kfd_doorbell_process_slice(pdd->dev); > } > diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c > b/drivers/gpu/drm/amd/amdkfd/kfd_process.c > index 6c83a519b3a1..951b63677248 100644 > --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c > +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c > @@ -1499,11 +1499,6 @@ struct kfd_process_device > *kfd_create_process_device_data(struct kfd_dev *dev, > if (!pdd) > return NULL; > > - if (kfd_alloc_process_doorbells(dev, &pdd->doorbell_index) < 0) { > - pr_err("Failed to alloc doorbell for pdd\n"); > - goto err_free_pdd; > - } > - > if (init_doorbell_bitmap(&pdd->qpd, dev)) { > pr_err("Failed to init doorbell for process\n"); > goto err_free_pdd; > -- > 2.32.0<o:p></o:p> </div> </div> </div> </body> </html>