[PATCH drm-misc-next 1/3] drm/sched: implement dynamic job flow control

[Luben Tuikov]

Hi,

On 2023-09-26 20:13, Danilo Krummrich wrote:
> On 9/26/23 22:43, Luben Tuikov wrote:
>> Hi,
>>
>> On 2023-09-24 18:43, Danilo Krummrich wrote:
>>> Currently, job flow control is implemented simply by limiting the amount
>>> of jobs in flight. Therefore, a scheduler is initialized with a
>>> submission limit that corresponds to a certain amount of jobs.
>>
>> "certain"? How about this instead:
>> " ... that corresponds to the number of jobs which can be sent
>>    to the hardware."?
>>
>>>
>>> This implies that for each job drivers need to account for the maximum
>>                                  ^,
>> Please add a comma after "job".
>>
>>> job size possible in order to not overflow the ring buffer.
>>
>> Well, different hardware designs would implement this differently.
>> Ideally, you only want pointers into the ring buffer, and then
>> the hardware consumes as much as it can. But this is a moot point
>> and it's always a good idea to have a "job size" hint from the client.
>> So this is a good patch.
>>
>> Ideally, you want to say that the hardware needs to be able to
>> accommodate the number of jobs which can fit in the hardware
>> queue times the largest job. This is a waste of resources
>> however, and it is better to give a hint as to the size of a job,
>> by the client. If the hardware can peek and understand dependencies,
>> on top of knowing the "size of the job", it can be an extremely
>> efficient scheduler.
>>
>>>
>>> However, there are drivers, such as Nouveau, where the job size has a
>>> rather large range. For such drivers it can easily happen that job
>>> submissions not even filling the ring by 1% can block subsequent
>>> submissions, which, in the worst case, can lead to the ring run dry.
>>>
>>> In order to overcome this issue, allow for tracking the actual job size
>>> instead of the amount job jobs. Therefore, add a field to track a job's
>>
>> "the amount job jobs." --> "the number of jobs."
> 
> Yeah, I somehow manage to always get this wrong, which I guess you noticed
> below already.
> 
> That's all good points below - gonna address them.

Forgot to mention, title tweak,
 "implement dynamic job flow control" --> "implement dynamic flow job control"
would perhaps be better? Unless that was meant to be "job-flow control"?

> Did you see Boris' response regarding a separate callback in order to fetch
> the job's submission units dynamically? Since this is needed by PowerVR, I'd
> like to include this in V2. What's your take on that?

Both of them have good valid points. The whole point is to guarantee forward
progress, and to be able to easily debug a stuck driver. Using a fence
in prepare-job would be easy to see that the fence never triggered, but
if we replace this with dynamic job credits, then debugging would be hard,
as Christian pointed out.

> My only concern with that would be that if I got what Boris was saying
> correctly calling
> 
> WARN_ON(s_job->submission_units > sched->submission_limit);
> 
> from drm_sched_can_queue() wouldn't work anymore, since this could indeed happen
> temporarily. I think this was also Christian's concern.

Indeed. We don't want hardware/drivers to game the scheduler, but want to
guarantee forward progress, e.g. a job with N number of credits completes,
and those credits are added to the available credits, and then a new,
smaller or bigger job is accepted for execution (prepare-job, run-job, etc.).

Feel free to rename "units" to "credits" as this is what is used in
hardware and link protocols, and naturally this is what I'm used to
calling such mechanisms.

I say, implement it, post the patch, and we'll take a look. It's a good thing
and we should definitely develop it.

Thanks!
-- 
Regards,
Luben