[Intel-gfx] [PATCH v4 5/5] drm/i915: Solve the GPU reset vs. modeset deadlocks with an rw_semaphore

[Daniel Vetter]

On Fri, Jun 30, 2017 at 04:53:19PM +0300, Ville Syrjälä wrote:
> On Fri, Jun 30, 2017 at 03:35:03PM +0200, Daniel Vetter wrote:
> > On Thu, Jun 29, 2017 at 10:26:08PM +0300, Ville Syrjälä wrote:
> > > On Thu, Jun 29, 2017 at 06:57:30PM +0100, Chris Wilson wrote:
> > > > Quoting ville.syrjala at linux.intel.com (2017-06-29 15:36:42)
> > > > > From: Ville Syrjälä <ville.syrjala at linux.intel.com>
> > > > > 
> > > > > Introduce an rw_semaphore to protect the display commits. All normal
> > > > > commits use down_read() and hence can proceed in parallel, but GPU reset
> > > > > will use down_write() making sure no other commits are in progress when
> > > > > we have to pull the plug on the display engine on pre-g4x platforms.
> > > > > There are no modeset/gem locks taken inside __intel_atomic_commit_tail()
> > > > > itself, and we wait for all dependencies before the down_read(), and
> > > > > thus there is no chance of deadlocks with this scheme.
> > > > 
> > > > This matches what I thought should be done (I didn't think of using
> > > > rwsem just a mutex, nice touch). The point I got stuck on was what
> > > > should be done after the reset? I expected another modeset to return the
> > > > state back or otherwise the inflight would get confused?
> > > 
> > > I guess that can happen. For instance, if we have a crtc_enable() in flight,
> > > and then we do a reset before it gets committed we would end up doing
> > > crtc_enable() twice in a row without a crtc_disable in between. For page
> > > flips and such this shouldn't be a big deal in general.
> > 
> > atomic commits are ordered. You have to wait for the previous ones to
> > complete before you do a new one. If you don't do that, then all hell
> > breaks loose.
> 
> What we're effectively doing here is inserting two new commits in
> the middle of the timeline, one to disable everything, and another
> one to re-enable everything. At the end of the the re-enable we would
> want the hardware state should match exactly what was there before
> the disable, hence any commits still in the timeline should work
> correctly. That is, their old_state will match the hardware state
> when it comes time to commit them.
> 
> But we can only do that properly after we start to track the committed
> state. Without that tracking we can get into trouble wrt. the
> hardware state not matching the old state when it comes time to
> commit the new state.

Yeah, but my point is that this here is an extremely fancy and fragile
(and afaics in this form, incomplete) fix for something that in the past
was fixed much, much simpler. Why do we need this massive amount of
complexity now? Who's asking for all this (we don't even have anyone yet
asking for fully queued atomic commits, much less on gen4)?

I mean rewriting the entire modeset code is fun and all, but for gen3-4?

> > What you really can't do with atomic (without rewriting everything once
> > more) is cancel a commit. Pre-atomic we could do that on gen4 since the
> > mmio flips died with the gpu, but that's the one design change we need to
> > cope with (plus TDR insisting it can't force-complete requests anymore).
> > 
> > > > > During reset we should be recommiting the state that was committed last.
> > > > > But for now we'll settle for recommiting the last state for each object.
> > > > 
> > > > Ah, I guess that explains the above. What is the complication with
> > > > restoring the current state as opposed to the next state?
> > > 
> > > Well the main thing is just tracking which is the current state. That
> > > just needs refactoring the .atomic_duplicate_state() calling convention
> > > across the whole tree so that we can then duplicate the committed state
> > > rather than the latest state.
> > > 
> > > Also due to the commit_hw_done() being potentially done after the
> > > modeset locks have been dropped, I don't think we can be certain
> > > of it getting called in the same order as swap_state(), hence
> > > when we track the committed state in commit_hw_done() we'll have
> > > to have some way to figure out if our new state is in fact the
> > > latest committed state for each object or if the calls got
> > > reordered. We don't insert any dependencies between two commits
> > > unless they touch the same active crtc, thus this reordering
> > > seems very much possible. Dunno if we should add some way to add
> > > such dependeencies whenever the same object is part of two otherwise
> > > independent commits, or if we just want to try and work with the
> > > reordered calls. My idea for the latter was some kind of seqno/age
> > > counter on the object states that allows me to recongnize which
> > > state is more recent. The object states aren't refcounted so hanging
> > > on to the wrong pointer could cause an oops the next time we have to
> > > perform a GPU reset.
> > 
> > Atomic commits are strongly ordered on a given CRTC, so stuff can't be
> > out-of-order within one. Across them the idea was to just add more CRTC
> > states into the atomic commit to make sure stuff is ordered correctly.
> 
> And atomic commits not touching the same crtc will not be ordered in any
> way. Thus if they touch the same object (eg. disabled plane or connector)
> we can't currently tell if the commit_hw_done() calls happened in the same
> order as the swap_state() calls for that particular object.

Yes, but I think for i915 it's ok, because we don't have planes that move
around (not supported at least), and for other shared stuff we just grab
them all. It is a bug in general though, and for i915 it would probably be
best if we'd add the various drm_crtc_commit waits to the i915_sw_fence.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch