[Intel-gfx] [PATCH 08/21] drm/i915/gem: Disallow bonding of virtual engines

[Matthew Brost]

On Thu, Apr 29, 2021 at 02:14:19PM +0200, Daniel Vetter wrote:
> On Wed, Apr 28, 2021 at 01:17:27PM -0500, Jason Ekstrand wrote:
> > On Wed, Apr 28, 2021 at 1:02 PM Matthew Brost <matthew.brost at intel.com> wrote:
> > >
> > > On Wed, Apr 28, 2021 at 12:46:07PM -0500, Jason Ekstrand wrote:
> > > > On Wed, Apr 28, 2021 at 12:26 PM Matthew Brost <matthew.brost at intel.com> wrote:
> > > > > Jumping on here mid-thread. For what is is worth to make execlists work
> > > > > with the upcoming parallel submission extension I leveraged some of the
> > > > > existing bonding code so I wouldn't be too eager to delete this code
> > > > > until that lands.
> > > >
> > > > Mind being a bit more specific about that?  The motivation for this
> > > > patch is that the current bonding handling and uAPI is, well, very odd
> > > > and confusing IMO.  It doesn't let you create sets of bonded engines.
> > > > Instead you create engines and then bond them together after the fact.
> > > > I didn't want to blindly duplicate those oddities with the proto-ctx
> > > > stuff unless they were useful.  With parallel submit, I would expect
> > > > we want a more explicit API where you specify a set of engine
> > > > class/instance pairs to bond together into a single engine similar to
> > > > how the current balancing API works.
> > > >
> > > > Of course, that's all focused on the API and not the internals.  But,
> > > > again, I'm not sure how we want things to look internally.  What we've
> > > > got now doesn't seem great for the GuC submission model but I'm very
> > > > much not the expert there.  I don't want to be working at cross
> > > > purposes to you and I'm happy to leave bits if you think they're
> > > > useful.  But I thought I was clearing things away so that you can put
> > > > in what you actually want for GuC/parallel submit.
> > > >
> > >
> > > Removing all the UAPI things are fine but I wouldn't delete some of the
> > > internal stuff (e.g. intel_virtual_engine_attach_bond, bond
> > > intel_context_ops, the hook for a submit fence, etc...) as that will
> > > still likely be used for the new parallel submission interface with
> > > execlists. As you say the new UAPI wont allow crazy configurations,
> > > only simple ones.
> > 
> > I'm fine with leaving some of the internal bits for a little while if
> > it makes pulling the GuC scheduler in easier.  I'm just a bit
> > skeptical of why you'd care about SUBMIT_FENCE. :-)  Daniel, any
> > thoughts?
> 
> Yeah I'm also wondering why we need this. Essentially your insight (and
> Tony Ye from media team confirmed) is that media umd never uses bonded on
> virtual engines.
>

Well you should use virtual engines with parallel submission interface 
if are you using it correctly.

e.g. You want a 2 wide parallel submission and there are 4 engine
instances.

You'd create 2 VEs:

A: 0, 2
B: 1, 3
set_parallel

For GuC submission we just configure context and the GuC load balances
it.

For execlists we'd need to create bonds.

Also likely the reason virtual engines wasn't used with the old
interface was we only had 2 instances max per class so no need for
virtual engines. If they used it for my above example if they were using
the interface correctly they would have to use virtual engines too.
 
> So the only thing we need is the await_fence submit_fence logic to stall
> the subsequent patches just long enough. I think that stays.
>

My implementation, for the new parallel submission interface, with
execlists used a bonds + priority boosts to ensure both are present at
the same time. This was used for both non-virtual and virtual engines.
This was never reviewed though and the code died on the list.

> All the additional logic with the cmpxchg lockless trickery and all that
> isn't needed, because we _never_ have to select an engine for bonded
> submission: It's always the single one available.
> 
> This would mean that for execlist parallel submit we can apply a
> limitation (beyond what GuC supports perhaps) and it's all ok. With that
> everything except the submit fence await logic itself can go I think.
> 
> Also one for Matt: We decided to ZBB implementing parallel submit on
> execlist, it's going to be just for GuC. At least until someone starts
> screaming really loudly.

If this is the case, then bonds can be deleted.

Matt

> 
> Cheers, Daniel
> -- 
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch