[Mesa-dev] Linux Graphics Next: Userspace submission update

[Daniel Vetter]

On Fri, Jun 04, 2021 at 01:27:15PM +0200, Christian König wrote:
> Am 04.06.21 um 10:57 schrieb Daniel Vetter:
> > On Fri, Jun 04, 2021 at 09:00:31AM +0200, Christian König wrote:
> > > Am 02.06.21 um 21:19 schrieb Daniel Vetter:
> > > > On Wed, Jun 02, 2021 at 08:52:38PM +0200, Christian König wrote:
> > > > > Am 02.06.21 um 20:48 schrieb Daniel Vetter:
> > > > > > On Wed, Jun 02, 2021 at 05:38:51AM -0400, Marek Olšák wrote:
> > > > > > > On Wed, Jun 2, 2021 at 5:34 AM Marek Olšák <maraeo at gmail.com> wrote:
> > > > > > > 
> > > > > > > > Yes, we can't break anything because we don't want to complicate things
> > > > > > > > for us. It's pretty much all NAK'd already. We are trying to gather more
> > > > > > > > knowledge and then make better decisions.
> > > > > > > > 
> > > > > > > > The idea we are considering is that we'll expose memory-based sync objects
> > > > > > > > to userspace for read only, and the kernel or hw will strictly control the
> > > > > > > > memory writes to those sync objects. The hole in that idea is that
> > > > > > > > userspace can decide not to signal a job, so even if userspace can't
> > > > > > > > overwrite memory-based sync object states arbitrarily, it can still decide
> > > > > > > > not to signal them, and then a future fence is born.
> > > > > > > > 
> > > > > > > This would actually be treated as a GPU hang caused by that context, so it
> > > > > > > should be fine.
> > > > > > This is practically what I proposed already, except your not doing it with
> > > > > > dma_fence. And on the memory fence side this also doesn't actually give
> > > > > > what you want for that compute model.
> > > > > > 
> > > > > > This seems like a bit a worst of both worlds approach to me? Tons of work
> > > > > > in the kernel to hide these not-dma_fence-but-almost, and still pain to
> > > > > > actually drive the hardware like it should be for compute or direct
> > > > > > display.
> > > > > > 
> > > > > > Also maybe I've missed it, but I didn't see any replies to my suggestion
> > > > > > how to fake the entire dma_fence stuff on top of new hw. Would be
> > > > > > interesting to know what doesn't work there instead of amd folks going of
> > > > > > into internal again and then coming back with another rfc from out of
> > > > > > nowhere :-)
> > > > > Well to be honest I would just push back on our hardware/firmware guys that
> > > > > we need to keep kernel queues forever before going down that route.
> > > > I looked again, and you said the model wont work because preemption is way
> > > > too slow, even when the context is idle.
> > > > 
> > > > I guess at that point I got maybe too fed up and just figured "not my
> > > > problem", but if preempt is too slow as the unload fence, you can do it
> > > > with pte removal and tlb shootdown too (that is hopefully not too slow,
> > > > otherwise your hw is just garbage and wont even be fast for direct submit
> > > > compute workloads).
> > > Have you seen that one here:
> > > https://www.spinics.net/lists/amd-gfx/msg63101.html :)
> > > 
> > > I've rejected it because I think polling for 6 seconds on a TLB flush which
> > > can block interrupts as well is just madness.
> > Hm but I thought you had like 2 tlb flush modes, the shitty one (with
> > retrying page faults) and the not so shitty one?
> 
> Yeah, we call this the lightweight and the heavyweight tlb flush.
> 
> The lighweight can be used when you are sure that you don't have any of the
> PTEs currently in flight in the 3D/DMA engine and you just need to
> invalidate the TLB.
> 
> The heavyweight must be used when you need to invalidate the TLB *AND* make
> sure that no concurrently operation moves new stuff into the TLB.
> 
> The problem is for this use case we have to use the heavyweight one.

Just for my own curiosity: So the lightweight flush is only for in-between
CS when you know access is idle? Or does that also not work if userspace
has a CS on a dma engine going at the same time because the tlb aren't
isolated enough between engines?
-Daniel

> > But yeah at that point I think you just have to bite one of the bullets.
> 
> Yeah, completely agree. We can choose which way we want to die, but it's
> certainly not going to be nice whatever we do.
> 
> > 
> > The thing is with hmm/userspace memory fence model this will be even
> > worse, because you will _have_ to do this tlb flush deep down in core mm
> > functions, so this is going to be userptr, but worse.
> > 
> > With the dma_resv/dma_fence bo memory management model you can at least
> > wrap that tlb flush into a dma_fence and push the waiting/pinging onto a
> > separate thread or something like that. If the hw really is that slow.
> > 
> > Somewhat aside: Personally I think that sriov needs to move over to the
> > compute model, i.e. indefinite timeouts, no tdr, because everything takes
> > too long. At least looking around sriov timeouts tend to be 10x bare
> > metal, across the board.
> > 
> > But for stuff like cloud gaming that's serious amounts of heavy lifting
> > since it brings us right back "the entire linux/android 3d stack is built
> > on top of dma_fence right now".
> > 
> > > > The only thing that you need to do when you use pte clearing + tlb
> > > > shootdown instad of preemption as the unload fence for buffers that get
> > > > moved is that if you get any gpu page fault, you don't serve that, but
> > > > instead treat it as a tdr and shot the context permanently.
> > > > 
> > > > So summarizing the model I proposed:
> > > > 
> > > > - you allow userspace to directly write into the ringbuffer, and also
> > > >     write the fences directly
> > > > 
> > > > - actual submit is done by the kernel, using drm/scheduler. The kernel
> > > >     blindly trusts userspace to set up everything else, and even just wraps
> > > >     dma_fences around the userspace memory fences.
> > > > 
> > > > - the only check is tdr. If a fence doesn't complete an tdr fires, a) the
> > > >     kernel shot the entire context and b) userspace recovers by setting up a
> > > >     new ringbuffer
> > > > 
> > > > - memory management is done using ttm only, you still need to supply the
> > > >     buffer list (ofc that list includes the always present ones, so CS will
> > > >     only get the list of special buffers like today). If you hw can't trun
> > > >     gpu page faults and you ever get one we pull up the same old solution:
> > > >     Kernel shots the entire context.
> > > > 
> > > >     The important thing is that from the gpu pov memory management works
> > > >     exactly like compute workload with direct submit, except that you just
> > > >     terminate the context on _any_ page fault, instead of only those that go
> > > >     somewhere where there's really no mapping and repair the others.
> > > > 
> > > >     Also I guess from reading the old thread this means you'd disable page
> > > >     fault retry because that is apparently also way too slow for anything.
> > > > 
> > > > - memory management uses an unload fence. That unload fences waits for all
> > > >     userspace memory fences (represented as dma_fence) to complete, with
> > > >     maybe some fudge to busy-spin until we've reached the actual end of the
> > > >     ringbuffer (maybe you have a IB tail there after the memory fence write,
> > > >     we have that on intel hw), and it waits for the memory to get
> > > >     "unloaded". This is either preemption, or pte clearing + tlb shootdown,
> > > >     or whatever else your hw provides which is a) used for dynamic memory
> > > >     management b) fast enough for actual memory management.
> > > > 
> > > > - any time a context dies we force-complete all it's pending fences,
> > > >     in-order ofc
> > > > 
> > > > So from hw pov this looks 99% like direct userspace submit, with the exact
> > > > same mappings, command sequences and everything else. The only difference
> > > > is that the rinbuffer head/tail updates happen from drm/scheduler, instead
> > > > of directly from userspace.
> > > > 
> > > > None of this stuff needs funny tricks where the kernel controls the
> > > > writes to memory fences, or where you need kernel ringbuffers, or anything
> > > > like thist. Userspace is allowed to do anything stupid, the rules are
> > > > guaranteed with:
> > > > 
> > > > - we rely on the hw isolation features to work, but _exactly_ like compute
> > > >     direct submit would too
> > > > 
> > > > - dying on any page fault captures memory management issues
> > > > 
> > > > - dying (without kernel recover, this is up to userspace if it cares) on
> > > >     any tdr makes sure fences complete still
> > > > 
> > > > > That syncfile and all that Android stuff isn't working out of the box with
> > > > > the new shiny user queue submission model (which in turn is mostly because
> > > > > of Windows) already raised some eyebrows here.
> > > > I think if you really want to make sure the current linux stack doesn't
> > > > break the _only_ option you have is provide a ctx mode that allows
> > > > dma_fence and drm/scheduler to be used like today.
> > > Yeah, but I still can just tell our hw/fw guys that we really really need to
> > > keep kernel queues or the whole Linux/Android infrastructure needs to get a
> > > compatibility layer like you describe above.
> > > 
> > > > For everything else it sounds you're a few years too late, because even
> > > > just huge kernel changes wont happen in time. Much less rewriting
> > > > userspace protocols.
> > > Seconded, question is rather if we are going to start migrating at some
> > > point or if we should keep pushing on our hw/fw guys.
> > So from what I'm hearing other hw might gain the sw compat layer too. Plus
> > I'm hoping that with the sw compat layer it'd be easier to smooth over
> > userspace to the new model (because there will be a long time where we
> > have to support both, maybe even with a runtime switch from userspace
> > memory fences to dma_fence kernel stuff).
> > 
> > But in the end it's up to you what makes more sense between sw work and
> > hw/fw work involved.
> 
> I'm currently entertaining my head a bit with the idea of implementing the
> HW scheduling on the CPU.
> 
> The only obstacle that I can really see is that we might need to unmap a
> page in the CPU page table when a queue becomes idle.
> 
> But apart from that it would give us the required functionality, just
> without the hardware scheduler.
> 
> Christian.
> 
> > -Daniel
> 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch