[Linaro-mm-sig] [PATCH] RFC: dma-fence: Document recoverable page fault implications

Daniel Vetter daniel.vetter at ffwll.ch
Tue Feb 9 14:08:26 UTC 2021 

On Tue, Feb 9, 2021 at 12:15 PM Felix Kuehling <felix.kuehling at amd.com> wrote:
> Am 2021-02-09 um 1:37 a.m. schrieb Daniel Vetter:
> > On Tue, Feb 9, 2021 at 4:13 AM Bas Nieuwenhuizen
> > <bas at basnieuwenhuizen.nl> wrote:
> >> On Thu, Jan 28, 2021 at 4:40 PM Felix Kuehling <felix.kuehling at amd.com> wrote:
> >>> Am 2021-01-28 um 2:39 a.m. schrieb Christian König:
> >>>> Am 27.01.21 um 23:00 schrieb Felix Kuehling:
> >>>>> Am 2021-01-27 um 7:16 a.m. schrieb Christian König:
> >>>>>> Am 27.01.21 um 13:11 schrieb Maarten Lankhorst:
> >>>>>>> Op 27-01-2021 om 01:22 schreef Felix Kuehling:
> >>>>>>>> Am 2021-01-21 um 2:40 p.m. schrieb Daniel Vetter:
> >>>>>>>>> Recently there was a fairly long thread about recoreable hardware
> >>>>>>>>> page
> >>>>>>>>> faults, how they can deadlock, and what to do about that.
> >>>>>>>>>
> >>>>>>>>> While the discussion is still fresh I figured good time to try and
> >>>>>>>>> document the conclusions a bit.
> >>>>>>>>>
> >>>>>>>>> References:
> >>>>>>>>> https://lore.kernel.org/dri-devel/20210107030127.20393-1-Felix.Kuehling@amd.com/
> >>>>>>>>>
> >>>>>>>>> Cc: Maarten Lankhorst <maarten.lankhorst at linux.intel.com>
> >>>>>>>>> Cc: Thomas Hellström <thomas.hellstrom at intel.com>
> >>>>>>>>> Cc: "Christian König" <christian.koenig at amd.com>
> >>>>>>>>> Cc: Jerome Glisse <jglisse at redhat.com>
> >>>>>>>>> Cc: Felix Kuehling <felix.kuehling at amd.com>
> >>>>>>>>> Signed-off-by: Daniel Vetter <daniel.vetter at intel.com>
> >>>>>>>>> Cc: Sumit Semwal <sumit.semwal at linaro.org>
> >>>>>>>>> Cc: linux-media at vger.kernel.org
> >>>>>>>>> Cc: linaro-mm-sig at lists.linaro.org
> >>>>>>>>> --
> >>>>>>>>> I'll be away next week, but figured I'll type this up quickly for
> >>>>>>>>> some
> >>>>>>>>> comments and to check whether I got this all roughly right.
> >>>>>>>>>
> >>>>>>>>> Critique very much wanted on this, so that we can make sure hw which
> >>>>>>>>> can't preempt (with pagefaults pending) like gfx10 has a clear
> >>>>>>>>> path to
> >>>>>>>>> support page faults in upstream. So anything I missed, got wrong or
> >>>>>>>>> like that would be good.
> >>>>>>>>> -Daniel
> >>>>>>>>> ---
> >>>>>>>>>    Documentation/driver-api/dma-buf.rst | 66
> >>>>>>>>> ++++++++++++++++++++++++++++
> >>>>>>>>>    1 file changed, 66 insertions(+)
> >>>>>>>>>
> >>>>>>>>> diff --git a/Documentation/driver-api/dma-buf.rst
> >>>>>>>>> b/Documentation/driver-api/dma-buf.rst
> >>>>>>>>> index a2133d69872c..e924c1e4f7a3 100644
> >>>>>>>>> --- a/Documentation/driver-api/dma-buf.rst
> >>>>>>>>> +++ b/Documentation/driver-api/dma-buf.rst
> >>>>>>>>> @@ -257,3 +257,69 @@ fences in the kernel. This means:
> >>>>>>>>>      userspace is allowed to use userspace fencing or long running
> >>>>>>>>> compute
> >>>>>>>>>      workloads. This also means no implicit fencing for shared
> >>>>>>>>> buffers in these
> >>>>>>>>>      cases.
> >>>>>>>>> +
> >>>>>>>>> +Recoverable Hardware Page Faults Implications
> >>>>>>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>>>>>>>> +
> >>>>>>>>> +Modern hardware supports recoverable page faults, which has a
> >>>>>>>>> lot of
> >>>>>>>>> +implications for DMA fences.
> >>>>>>>>> +
> >>>>>>>>> +First, a pending page fault obviously holds up the work that's
> >>>>>>>>> running on the
> >>>>>>>>> +accelerator and a memory allocation is usually required to resolve
> >>>>>>>>> the fault.
> >>>>>>>>> +But memory allocations are not allowed to gate completion of DMA
> >>>>>>>>> fences, which
> >>>>>>>>> +means any workload using recoverable page faults cannot use DMA
> >>>>>>>>> fences for
> >>>>>>>>> +synchronization. Synchronization fences controlled by userspace
> >>>>>>>>> must be used
> >>>>>>>>> +instead.
> >>>>>>>>> +
> >>>>>>>>> +On GPUs this poses a problem, because current desktop compositor
> >>>>>>>>> protocols on
> >>>>>>>>> +Linus rely on DMA fences, which means without an entirely new
> >>>>>>>>> userspace stack
> >>>>>>>>> +built on top of userspace fences, they cannot benefit from
> >>>>>>>>> recoverable page
> >>>>>>>>> +faults. The exception is when page faults are only used as
> >>>>>>>>> migration hints and
> >>>>>>>>> +never to on-demand fill a memory request. For now this means
> >>>>>>>>> recoverable page
> >>>>>>>>> +faults on GPUs are limited to pure compute workloads.
> >>>>>>>>> +
> >>>>>>>>> +Furthermore GPUs usually have shared resources between the 3D
> >>>>>>>>> rendering and
> >>>>>>>>> +compute side, like compute units or command submission engines. If
> >>>>>>>>> both a 3D
> >>>>>>>>> +job with a DMA fence and a compute workload using recoverable page
> >>>>>>>>> faults are
> >>>>>>>>> +pending they could deadlock:
> >>>>>>>>> +
> >>>>>>>>> +- The 3D workload might need to wait for the compute job to finish
> >>>>>>>>> and release
> >>>>>>>>> +  hardware resources first.
> >>>>>>>>> +
> >>>>>>>>> +- The compute workload might be stuck in a page fault, because the
> >>>>>>>>> memory
> >>>>>>>>> +  allocation is waiting for the DMA fence of the 3D workload to
> >>>>>>>>> complete.
> >>>>>>>>> +
> >>>>>>>>> +There are a few ways to prevent this problem:
> >>>>>>>>> +
> >>>>>>>>> +- Compute workloads can always be preempted, even when a page
> >>>>>>>>> fault is pending
> >>>>>>>>> +  and not yet repaired. Not all hardware supports this.
> >>>>>>>>> +
> >>>>>>>>> +- DMA fence workloads and workloads which need page fault handling
> >>>>>>>>> have
> >>>>>>>>> +  independent hardware resources to guarantee forward progress.
> >>>>>>>>> This could be
> >>>>>>>>> +  achieved through e.g. through dedicated engines and minimal
> >>>>>>>>> compute unit
> >>>>>>>>> +  reservations for DMA fence workloads.
> >>>>>>>>> +
> >>>>>>>>> +- The reservation approach could be further refined by only
> >>>>>>>>> reserving the
> >>>>>>>>> +  hardware resources for DMA fence workloads when they are
> >>>>>>>>> in-flight. This must
> >>>>>>>>> +  cover the time from when the DMA fence is visible to other
> >>>>>>>>> threads up to
> >>>>>>>>> +  moment when fence is completed through dma_fence_signal().
> >>>>>>>>> +
> >>>>>>>>> +- As a last resort, if the hardware provides no useful reservation
> >>>>>>>>> mechanics,
> >>>>>>>>> +  all workloads must be flushed from the GPU when switching
> >>>>>>>>> between jobs
> >>>>>>>>> +  requiring DMA fences or jobs requiring page fault handling: This
> >>>>>>>>> means all DMA
> >>>>>>>>> +  fences must complete before a compute job with page fault
> >>>>>>>>> handling can be
> >>>>>>>>> +  inserted into the scheduler queue. And vice versa, before a DMA
> >>>>>>>>> fence can be
> >>>>>>>>> +  made visible anywhere in the system, all compute workloads must
> >>>>>>>>> be preempted
> >>>>>>>>> +  to guarantee all pending GPU page faults are flushed.
> >>>>>>>> I thought of another possible workaround:
> >>>>>>>>
> >>>>>>>>     * Partition the memory. Servicing of page faults will use a
> >>>>>>>> separate
> >>>>>>>>       memory pool that can always be allocated from without
> >>>>>>>> waiting for
> >>>>>>>>       fences. This includes memory for page tables and memory for
> >>>>>>>>       migrating data to. You may steal memory from other processes
> >>>>>>>> that
> >>>>>>>>       can page fault, so no fence waiting is necessary. Being able to
> >>>>>>>>       steal memory at any time also means there are basically no
> >>>>>>>>       out-of-memory situations you need to worry about. Even page
> >>>>>>>> tables
> >>>>>>>>       (except the root page directory of each process) can be
> >>>>>>>> stolen in
> >>>>>>>>       the worst case.
> >>>>>>> I think 'overcommit' would be a nice way to describe this. But I'm not
> >>>>>>> sure how easy this is to implement in practice. You would basically
> >>>>>>> need
> >>>>>>> to create your own memory manager for this.
> >>>>>> Well you would need a completely separate pool for both device as well
> >>>>>> as system memory.
> >>>>>>
> >>>>>> E.g. on boot we say we steal X GB system memory only for HMM.
> >>>>> Why? The GPU driver doesn't need to allocate system memory for HMM.
> >>>>> Migrations to system memory are handled by the kernel's handle_mm_fault
> >>>>> and page allocator and swap logic.
> >>>> And that one depends on dma_fence completion because you can easily
> >>>> need to wait for an MMU notifier callback.
> >>> I see, the GFX MMU notifier for userpointers in amdgpu currently waits
> >>> for fences. For the KFD MMU notifier I am planning to fix this by
> >>> causing GPU page faults instead of preempting the queues. Can we limit
> >>> userptrs in amdgpu to engines that can page fault. Basically make it
> >>> illegal to attach userptr BOs to graphics CS BO lists, so they can only
> >>> be used in user mode command submissions, which can page fault. Then the
> >>> GFX MMU notifier could invalidate PTEs and would not have to wait for
> >>> fences.
> >> sadly graphics + userptr is already exposed via Mesa.
> > This is not about userptr, we fake userptr entirely in software. It's
> > about exposing recoverable gpu page faults (which would make userptr
> > maybe more efficient since we could do on-demand paging). userptr
> > itself isn't a problem, but it is part of the reasons why this is
> > tricky.
> >
> > Christian/Felix, I think for kernel folks this is clear enough that I
> > don't need to clarify this in the text?
>
> Yeah, it's clear to me. Anyway, your latest text doesn't reference
> userptr directly and keeps the discussion at a fairly abstract level. So
> I think it's fine. It's the practical details of the proposed
> workarounds where it feel like walking through a mirror cabinet, bumping
> into unexpected obstacles with every other step.

Oh yes, this is very high-level. The implementation is going to be
very trick, no matter which one we're picking. And tbh I expect
surprises and things we'll learn. But I'm still hoping that this high
level doc patch will help a lot with avoiding the worst problems.

Of course once we have some of these hacks landed we should look at it
again and maybe update where it's wrong/unclear/...

btw r-b: from you too on the patch?

Cheers, Daniel

>
> Regards,
>   Felix
>
>
> > -Daniel
> >
> >>>
> >>>> As Maarten wrote when you want to go down this route you need a
> >>>> complete separate memory management parallel to the one of the kernel.
> >>> Not really. I'm trying to make the GPU memory management more similar to
> >>> what the kernel does for system memory.
> >>>
> >>> I understood Maarten's comment as "I'm creating a new memory manager and
> >>> not using TTM any more". This is true. The idea is that this portion of
> >>> VRAM would be managed more like system memory.
> >>>
> >>> Regards,
> >>>   Felix
> >>>
> >>>
> >>>> Regards,
> >>>> Christian.
> >>>>
> >>>>>   It doesn't depend on any fences, so
> >>>>> it cannot deadlock with any GPU driver-managed memory. The GPU driver
> >>>>> gets involved in the MMU notifier to invalidate device page tables. But
> >>>>> that also doesn't need to wait for any fences.
> >>>>>
> >>>>> And if the kernel runs out of pageable memory, you're in trouble anyway.
> >>>>> The OOM killer will step in, nothing new there.
> >>>>>
> >>>>> Regards,
> >>>>>    Felix
> >>>>>
> >>>>>
> >>>>>>> But from a design point of view, definitely a valid solution.
> >>>>>> I think the restriction above makes it pretty much unusable.
> >>>>>>
> >>>>>>> But this looks good, those solutions are definitely the valid
> >>>>>>> options we
> >>>>>>> can choose from.
> >>>>>> It's certainly worth noting, yes. And just to make sure that nobody
> >>>>>> has the idea to reserve only device memory.
> >>>>>>
> >>>>>> Christian.
> >>>>>>
> >>>>>>> ~Maarten
> >>>>>>>
> >>>>> _______________________________________________
> >>>>> Linaro-mm-sig mailing list
> >>>>> Linaro-mm-sig at lists.linaro.org
> >>>>> https://lists.linaro.org/mailman/listinfo/linaro-mm-sig
> >>>>>
> >>> _______________________________________________
> >>> dri-devel mailing list
> >>> dri-devel at lists.freedesktop.org
> >>> https://lists.freedesktop.org/mailman/listinfo/dri-devel
> >
> >



-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

More information about the dri-devel mailing list