Deadlock on PTEs update for HMM

[Sierra Guiza, Alejandro (Alex)]

Hi Christian,
As you know, we're working on the HMM enablement. Im working on the dGPU page table entries invalidation on the userptr mapping case. Currently, the MMU notifiers handle stops all user mode queues, schedule a delayed worker to re-validate userptr mappings and restart the queues.
Part of the HMM functionality, we need to invalidate the page table entries instead of stopping the queues. At the same time we need to move the revalidation of the userptr mappings into the page fault handler.
We're seeing a deadlock warning after we try to invalidate the PTEs inside the MMU notifier handler. More specific, when we try to update the BOs to invalidate PTEs using amdgpu_vm_bo_update. This uses kmalloc on the amdgpu_job_alloc which seems to be causing this problem.
Based on @Kuehling, Felix<mailto:Felix.Kuehling at amd.com> comments, kmalloc without any special flags can cause memory reclaim. Doing that inside an MMU notifier is problematic, because an MMU notifier may be called inside a memory-reclaim operation itself. That would result in recursion. Also, reclaim shouldn't be done while holding a lock that can be taken in an MMU notifier for the same reason. If you cause a reclaim while holding that lock, then an MMU notifier called by the reclaim can deadlock trying to take the same lock.
Please let us know if you have any advice to enable this the right way

Thanks in advanced,
Alejandro

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/amd-gfx/attachments/20191127/5d2fa875/attachment.html>