[Intel-gfx] [PATCH 08/21] drm/i915/gem: Disallow bonding of virtual engines

Sat May 1 17:17:46 UTC 2021

On Fri, Apr 30, 2021 at 12:11:07PM +0200, Daniel Vetter wrote:
> On Thu, Apr 29, 2021 at 09:03:48PM -0700, Matthew Brost wrote:
> > On Thu, Apr 29, 2021 at 02:14:19PM +0200, Daniel Vetter wrote:
> > > On Wed, Apr 28, 2021 at 01:17:27PM -0500, Jason Ekstrand wrote:
> > > > On Wed, Apr 28, 2021 at 1:02 PM Matthew Brost <matthew.brost at intel.com> wrote:
> > > > >
> > > > > On Wed, Apr 28, 2021 at 12:46:07PM -0500, Jason Ekstrand wrote:
> > > > > > On Wed, Apr 28, 2021 at 12:26 PM Matthew Brost <matthew.brost at intel.com> wrote:
> > > > > > > Jumping on here mid-thread. For what is is worth to make execlists work
> > > > > > > with the upcoming parallel submission extension I leveraged some of the
> > > > > > > existing bonding code so I wouldn't be too eager to delete this code
> > > > > > > until that lands.
> > > > > >
> > > > > > Mind being a bit more specific about that?  The motivation for this
> > > > > > patch is that the current bonding handling and uAPI is, well, very odd
> > > > > > and confusing IMO.  It doesn't let you create sets of bonded engines.
> > > > > > Instead you create engines and then bond them together after the fact.
> > > > > > I didn't want to blindly duplicate those oddities with the proto-ctx
> > > > > > stuff unless they were useful.  With parallel submit, I would expect
> > > > > > we want a more explicit API where you specify a set of engine
> > > > > > class/instance pairs to bond together into a single engine similar to
> > > > > > how the current balancing API works.
> > > > > >
> > > > > > Of course, that's all focused on the API and not the internals.  But,
> > > > > > again, I'm not sure how we want things to look internally.  What we've
> > > > > > got now doesn't seem great for the GuC submission model but I'm very
> > > > > > much not the expert there.  I don't want to be working at cross
> > > > > > purposes to you and I'm happy to leave bits if you think they're
> > > > > > useful.  But I thought I was clearing things away so that you can put
> > > > > > in what you actually want for GuC/parallel submit.
> > > > > >
> > > > >
> > > > > Removing all the UAPI things are fine but I wouldn't delete some of the
> > > > > internal stuff (e.g. intel_virtual_engine_attach_bond, bond
> > > > > intel_context_ops, the hook for a submit fence, etc...) as that will
> > > > > still likely be used for the new parallel submission interface with
> > > > > execlists. As you say the new UAPI wont allow crazy configurations,
> > > > > only simple ones.
> > > > 
> > > > I'm fine with leaving some of the internal bits for a little while if
> > > > it makes pulling the GuC scheduler in easier.  I'm just a bit
> > > > skeptical of why you'd care about SUBMIT_FENCE. :-)  Daniel, any
> > > > thoughts?
> > > 
> > > Yeah I'm also wondering why we need this. Essentially your insight (and
> > > Tony Ye from media team confirmed) is that media umd never uses bonded on
> > > virtual engines.
> > >
> > 
> > Well you should use virtual engines with parallel submission interface 
> > if are you using it correctly.
> > 
> > e.g. You want a 2 wide parallel submission and there are 4 engine
> > instances.
> > 
> > You'd create 2 VEs:
> > 
> > A: 0, 2
> > B: 1, 3
> > set_parallel
> 
> So tbh I'm not really liking this part. At least my understanding is that
> with GuC this is really one overall virtual engine, backed by a multi-lrc.
> 
> So it should fill one engine slot, not fill multiple virtual engines and
> then be an awkward thing wrapped on top.
> 
> I think (but maybe my understanding of GuC and the parallel submit execbuf
> interface is wrong) that the parallel engine should occupy a single VE
> slot, not require additional VE just for fun (maybe the execlist backend
> would require that internally, but that should not leak into the higher
> levels, much less the uapi). And you submit your multi-batch execbuf on
> that single parallel VE, which then gets passed to GuC as a multi-LRC.
> Internally in the backend there's a bit of fan-out to put the right
> MI_BB_START into the right rings and all that, but again I think that
> should be backend concerns.
> 
> Or am I missing something big here?

Unfortunately that is not how the interface works. The user must
configure the engine set with either physical or virtual engines which
determine the valid placements of each BB (LRC, ring, whatever we want
to call it) and call the set parallel extension which validations engine
layout. After that the engines are ready be used with multi-BB
submission in single IOCTL. 

We discussed this internally with the i915 developers + with the media
for like 6 months and this is where we landed after some very
contentious discussions. One of the proposals was pretty similar to your
understanding but got NACK'd as it was too specific to what our HW can
do / what the UMDs need rather than being able to do tons of wild things
our HW / UMDs will never support (sounds familiar, right?). 

What we landed on is still simpler than most of the other proposals - we
almost really went off the deep end but voices of reason thankfully won
out.

> 
> > For GuC submission we just configure context and the GuC load balances
> > it.
> > 
> > For execlists we'd need to create bonds.
> > 
> > Also likely the reason virtual engines wasn't used with the old
> > interface was we only had 2 instances max per class so no need for
> > virtual engines. If they used it for my above example if they were using
> > the interface correctly they would have to use virtual engines too.
> 
> They do actually use virtual engines, it's just the virtual engine only
> contains a single one, and internally i915 folds that into the hw engine
> directly. So we can take away the entire implementation complexity.
> 
> Also I still think for execlist we shouldn't bother with trying to enable
> parallel submit. Or at least only way down if there's no other reasonable
> option.
>

Agree but honestly if we have to it isn't going to be that painful. I
think my patch to enable this was a couple hundred lines.

Matt

> > > So the only thing we need is the await_fence submit_fence logic to stall
> > > the subsequent patches just long enough. I think that stays.
> > >
> > 
> > My implementation, for the new parallel submission interface, with
> > execlists used a bonds + priority boosts to ensure both are present at
> > the same time. This was used for both non-virtual and virtual engines.
> > This was never reviewed though and the code died on the list.
> 
> :-(
> 
> > > All the additional logic with the cmpxchg lockless trickery and all that
> > > isn't needed, because we _never_ have to select an engine for bonded
> > > submission: It's always the single one available.
> > > 
> > > This would mean that for execlist parallel submit we can apply a
> > > limitation (beyond what GuC supports perhaps) and it's all ok. With that
> > > everything except the submit fence await logic itself can go I think.
> > > 
> > > Also one for Matt: We decided to ZBB implementing parallel submit on
> > > execlist, it's going to be just for GuC. At least until someone starts
> > > screaming really loudly.
> > 
> > If this is the case, then bonds can be deleted.
> 
> Yeah that's the goal we're aiming for.
> -Daniel
> -- 
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch