[Intel-gfx] [PATCH 08/21] drm/i915/gem: Disallow bonding of virtual engines

[Daniel Vetter]

On Thu, Apr 29, 2021 at 09:03:48PM -0700, Matthew Brost wrote:
> On Thu, Apr 29, 2021 at 02:14:19PM +0200, Daniel Vetter wrote:
> > On Wed, Apr 28, 2021 at 01:17:27PM -0500, Jason Ekstrand wrote:
> > > On Wed, Apr 28, 2021 at 1:02 PM Matthew Brost <matthew.brost at intel.com> wrote:
> > > >
> > > > On Wed, Apr 28, 2021 at 12:46:07PM -0500, Jason Ekstrand wrote:
> > > > > On Wed, Apr 28, 2021 at 12:26 PM Matthew Brost <matthew.brost at intel.com> wrote:
> > > > > > Jumping on here mid-thread. For what is is worth to make execlists work
> > > > > > with the upcoming parallel submission extension I leveraged some of the
> > > > > > existing bonding code so I wouldn't be too eager to delete this code
> > > > > > until that lands.
> > > > >
> > > > > Mind being a bit more specific about that?  The motivation for this
> > > > > patch is that the current bonding handling and uAPI is, well, very odd
> > > > > and confusing IMO.  It doesn't let you create sets of bonded engines.
> > > > > Instead you create engines and then bond them together after the fact.
> > > > > I didn't want to blindly duplicate those oddities with the proto-ctx
> > > > > stuff unless they were useful.  With parallel submit, I would expect
> > > > > we want a more explicit API where you specify a set of engine
> > > > > class/instance pairs to bond together into a single engine similar to
> > > > > how the current balancing API works.
> > > > >
> > > > > Of course, that's all focused on the API and not the internals.  But,
> > > > > again, I'm not sure how we want things to look internally.  What we've
> > > > > got now doesn't seem great for the GuC submission model but I'm very
> > > > > much not the expert there.  I don't want to be working at cross
> > > > > purposes to you and I'm happy to leave bits if you think they're
> > > > > useful.  But I thought I was clearing things away so that you can put
> > > > > in what you actually want for GuC/parallel submit.
> > > > >
> > > >
> > > > Removing all the UAPI things are fine but I wouldn't delete some of the
> > > > internal stuff (e.g. intel_virtual_engine_attach_bond, bond
> > > > intel_context_ops, the hook for a submit fence, etc...) as that will
> > > > still likely be used for the new parallel submission interface with
> > > > execlists. As you say the new UAPI wont allow crazy configurations,
> > > > only simple ones.
> > > 
> > > I'm fine with leaving some of the internal bits for a little while if
> > > it makes pulling the GuC scheduler in easier.  I'm just a bit
> > > skeptical of why you'd care about SUBMIT_FENCE. :-)  Daniel, any
> > > thoughts?
> > 
> > Yeah I'm also wondering why we need this. Essentially your insight (and
> > Tony Ye from media team confirmed) is that media umd never uses bonded on
> > virtual engines.
> >
> 
> Well you should use virtual engines with parallel submission interface 
> if are you using it correctly.
> 
> e.g. You want a 2 wide parallel submission and there are 4 engine
> instances.
> 
> You'd create 2 VEs:
> 
> A: 0, 2
> B: 1, 3
> set_parallel

So tbh I'm not really liking this part. At least my understanding is that
with GuC this is really one overall virtual engine, backed by a multi-lrc.

So it should fill one engine slot, not fill multiple virtual engines and
then be an awkward thing wrapped on top.

I think (but maybe my understanding of GuC and the parallel submit execbuf
interface is wrong) that the parallel engine should occupy a single VE
slot, not require additional VE just for fun (maybe the execlist backend
would require that internally, but that should not leak into the higher
levels, much less the uapi). And you submit your multi-batch execbuf on
that single parallel VE, which then gets passed to GuC as a multi-LRC.
Internally in the backend there's a bit of fan-out to put the right
MI_BB_START into the right rings and all that, but again I think that
should be backend concerns.

Or am I missing something big here?

> For GuC submission we just configure context and the GuC load balances
> it.
> 
> For execlists we'd need to create bonds.
> 
> Also likely the reason virtual engines wasn't used with the old
> interface was we only had 2 instances max per class so no need for
> virtual engines. If they used it for my above example if they were using
> the interface correctly they would have to use virtual engines too.

They do actually use virtual engines, it's just the virtual engine only
contains a single one, and internally i915 folds that into the hw engine
directly. So we can take away the entire implementation complexity.

Also I still think for execlist we shouldn't bother with trying to enable
parallel submit. Or at least only way down if there's no other reasonable
option.

> > So the only thing we need is the await_fence submit_fence logic to stall
> > the subsequent patches just long enough. I think that stays.
> >
> 
> My implementation, for the new parallel submission interface, with
> execlists used a bonds + priority boosts to ensure both are present at
> the same time. This was used for both non-virtual and virtual engines.
> This was never reviewed though and the code died on the list.

:-(

> > All the additional logic with the cmpxchg lockless trickery and all that
> > isn't needed, because we _never_ have to select an engine for bonded
> > submission: It's always the single one available.
> > 
> > This would mean that for execlist parallel submit we can apply a
> > limitation (beyond what GuC supports perhaps) and it's all ok. With that
> > everything except the submit fence await logic itself can go I think.
> > 
> > Also one for Matt: We decided to ZBB implementing parallel submit on
> > execlist, it's going to be just for GuC. At least until someone starts
> > screaming really loudly.
> 
> If this is the case, then bonds can be deleted.

Yeah that's the goal we're aiming for.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch