[PATCH] drm/amdgpu: set sched_hw_submission higher for KIQ

[Jay Cornwall]

On Tue, Aug 22, 2017, at 16:17, Felix Kuehling wrote:
> Thanks Alex!
> 
> Jay, do you think this is enough? This bumps the number of concurrent
> operations on KIQ to 4 by default.

I'm not sure what the best number is. Up to 8 KFD processes is common
(beyond that performance drops off due to VMID availability) but I'm not
sure how often they would need to submit to KIQ concurrently. If it's
not expensive I'd just bump it up to say 16.

The performance problem isn't that bad since all the KIQ requests are
serialized but the dmesg spam is not nice. Perhaps lowering the severity
of the 'rcu slot is busy' message would address that as well?

> 
> Regards,
>   Felix
> 
> 
> On 2017-08-22 04:49 PM, Alex Deucher wrote:
> > KIQ doesn't really use the GPU scheduler.  The base
> > drivers generally use the KIQ ring directly rather than
> > submitting IBs.  However, amdgpu_sched_hw_submission
> > (which defaults to 2) limits the number of outstanding
> > fences to 2.  KFD uses the KIQ for TLB flushes and the
> > 2 fence limit hurts performance when there are several KFD
> > processes running.
> >
> > Signed-off-by: Alex Deucher <alexander.deucher at amd.com>
> > ---
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 14 ++++++++++++--
> >  1 file changed, 12 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> > index 6c5646b..f39b851 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
> > @@ -170,6 +170,16 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
> >  		     unsigned irq_type)
> >  {
> >  	int r;
> > +	int sched_hw_submission = amdgpu_sched_hw_submission;
> > +
> > +	/* Set the hw submission limit higher for KIQ because
> > +	 * it's used for a number of gfx/compute tasks by both
> > +	 * KFD and KGD which may have outstanding fences and
> > +	 * it doesn't really use the gpu scheduler anyway;
> > +	 * KIQ tasks get submitted directly to the ring.
> > +	 */
> > +	if (ring->funcs->type == AMDGPU_RING_TYPE_KIQ)
> > +		sched_hw_submission *= 2;
> >  
> >  	if (ring->adev == NULL) {
> >  		if (adev->num_rings >= AMDGPU_MAX_RINGS)
> > @@ -179,7 +189,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
> >  		ring->idx = adev->num_rings++;
> >  		adev->rings[ring->idx] = ring;
> >  		r = amdgpu_fence_driver_init_ring(ring,
> > -			amdgpu_sched_hw_submission);
> > +						  sched_hw_submission);
> >  		if (r)
> >  			return r;
> >  	}
> > @@ -219,7 +229,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
> >  	}
> >  
> >  	ring->ring_size = roundup_pow_of_two(max_dw * 4 *
> > -					     amdgpu_sched_hw_submission);
> > +					     sched_hw_submission);
> >  
> >  	ring->buf_mask = (ring->ring_size / 4) - 1;
> >  	ring->ptr_mask = ring->funcs->support_64bit_ptrs ?
> 
> _______________________________________________
> amd-gfx mailing list
> amd-gfx at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx