[PATCH 1/2] drm/scheduler: improve GPU scheduler documentation v2

[Christian König]

Am 19.07.24 um 12:29 schrieb Philipp Stanner:
> On Thu, 2023-11-16 at 15:15 +0100, Christian König wrote:
>> Start to improve the scheduler document. Especially document the
>> lifetime of each of the objects as well as the restrictions around
>> DMA-fence handling and userspace compatibility.
> Hallo Christian,
>
> thanks for working on this.
>
> I'm currently looking deeper into the scheduler and am documenting the
> pitfalls etc. that I have found so far.
>
>
> What are your current plans with this documentation series? If you
> don't intend to get it upstreamed in the foreseeable future, I would
> like to hijack the series and use it as a basis for my own improvements
> to the documentation.
>
> Please tell me what you think,

Feel free to pick that up, I have around 10 different upstream things on 
my backlog at the moment and it didn't got any smaller since the 
beginning of the year :(

I really need to sort out my time management to be able to work on 
upstream stuff without getting interrupted by anything AMD internal.

Cheers,
Christian.

>
>
> Regards,
> P.
>
>
>> v2: Some improvements suggested by Danilo, add section about error
>>      handling.
>>
>> Signed-off-by: Christian König <christian.koenig at amd.com>
>> ---
>>   Documentation/gpu/drm-mm.rst           |  36 +++++
>>   drivers/gpu/drm/scheduler/sched_main.c | 174 +++++++++++++++++++++--
>> --
>>   2 files changed, 188 insertions(+), 22 deletions(-)
>>
>> diff --git a/Documentation/gpu/drm-mm.rst b/Documentation/gpu/drm-
>> mm.rst
>> index acc5901ac840..112463fa9f3a 100644
>> --- a/Documentation/gpu/drm-mm.rst
>> +++ b/Documentation/gpu/drm-mm.rst
>> @@ -552,12 +552,48 @@ Overview
>>   .. kernel-doc:: drivers/gpu/drm/scheduler/sched_main.c
>>      :doc: Overview
>>   
>> +Job Object
>> +----------
>> +
>> +.. kernel-doc:: drivers/gpu/drm/scheduler/sched_main.c
>> +   :doc: Job Object
>> +
>> +Entity Object
>> +-------------
>> +
>> +.. kernel-doc:: drivers/gpu/drm/scheduler/sched_main.c
>> +   :doc: Entity Object
>> +
>> +Hardware Fence Object
>> +---------------------
>> +
>> +.. kernel-doc:: drivers/gpu/drm/scheduler/sched_main.c
>> +   :doc: Hardware Fence Object
>> +
>> +Scheduler Fence Object
>> +----------------------
>> +
>> +.. kernel-doc:: drivers/gpu/drm/scheduler/sched_main.c
>> +   :doc: Scheduler Fence Object
>> +
>> +Scheduler and Run Queue Objects
>> +-------------------------------
>> +
>> +.. kernel-doc:: drivers/gpu/drm/scheduler/sched_main.c
>> +   :doc: Scheduler and Run Queue Objects
>> +
>>   Flow Control
>>   ------------
>>   
>>   .. kernel-doc:: drivers/gpu/drm/scheduler/sched_main.c
>>      :doc: Flow Control
>>   
>> +Error and Timeout handling
>> +--------------------------
>> +
>> +.. kernel-doc:: drivers/gpu/drm/scheduler/sched_main.c
>> +   :doc: Error and Timeout handling
>> +
>>   Scheduler Function References
>>   -----------------------------
>>   
>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c
>> b/drivers/gpu/drm/scheduler/sched_main.c
>> index 044a8c4875ba..026123497b0e 100644
>> --- a/drivers/gpu/drm/scheduler/sched_main.c
>> +++ b/drivers/gpu/drm/scheduler/sched_main.c
>> @@ -24,28 +24,122 @@
>>   /**
>>    * DOC: Overview
>>    *
>> - * The GPU scheduler provides entities which allow userspace to push
>> jobs
>> - * into software queues which are then scheduled on a hardware run
>> queue.
>> - * The software queues have a priority among them. The scheduler
>> selects the entities
>> - * from the run queue using a FIFO. The scheduler provides
>> dependency handling
>> - * features among jobs. The driver is supposed to provide callback
>> functions for
>> - * backend operations to the scheduler like submitting a job to
>> hardware run queue,
>> - * returning the dependencies of a job etc.
>> - *
>> - * The organisation of the scheduler is the following:
>> - *
>> - * 1. Each hw run queue has one scheduler
>> - * 2. Each scheduler has multiple run queues with different
>> priorities
>> - *    (e.g., HIGH_HW,HIGH_SW, KERNEL, NORMAL)
>> - * 3. Each scheduler run queue has a queue of entities to schedule
>> - * 4. Entities themselves maintain a queue of jobs that will be
>> scheduled on
>> - *    the hardware.
>> - *
>> - * The jobs in a entity are always scheduled in the order that they
>> were pushed.
>> - *
>> - * Note that once a job was taken from the entities queue and pushed
>> to the
>> - * hardware, i.e. the pending queue, the entity must not be
>> referenced anymore
>> - * through the jobs entity pointer.
>> + * The GPU scheduler implements some logic to decide which command
>> submission
>> + * to push next to the hardware. Another major use case of the GPU
>> scheduler
>> + * is to enforce correct driver behavior around those command
>> submissions.
>> + * Because of this it's also used by drivers which don't need the
>> actual
>> + * scheduling functionality.
>> + *
>> + * All callbacks the driver needs to implement are restricted by
>> DMA-fence
>> + * signaling rules to guarantee deadlock free forward progress. This
>> especially
>> + * means that for normal operation no memory can be allocated in a
>> callback.
>> + * All memory which is needed for pushing the job to the hardware
>> must be
>> + * allocated before arming a job. It also means that no locks can be
>> taken
>> + * under which memory might be allocated as well.
>> + *
>> + * Memory which is optional to allocate, for example for device core
>> dumping or
>> + * debugging, *must* be allocated with GFP_NOWAIT and appropriate
>> error
>> + * handling taking if that allocation fails. GFP_ATOMIC should only
>> be used if
>> + * absolutely necessary since dipping into the special atomic
>> reserves is
>> + * usually not justified for a GPU driver.
>> + */
>> +
>> +/**
>> + * DOC: Job Object
>> + *
>> + * The base job object contains submission dependencies in the form
>> of DMA-fence
>> + * objects. Drivers can also implement an optional prepare_job
>> callback which
>> + * returns additional dependencies as DMA-fence objects. It's
>> important to note
>> + * that this callback can't allocate memory or grab locks under
>> which memory is
>> + * allocated.
>> + *
>> + * Drivers should use this as base class for an object which
>> contains the
>> + * necessary state to push the command submission to the hardware.
>> + *
>> + * The lifetime of the job object should at least be from pushing it
>> into the
>> + * scheduler until the scheduler notes through the free callback
>> that a job
>> + * isn't needed any more. Drivers can of course keep their job
>> object alive
>> + * longer than that, but that's outside of the scope of the
>> scheduler
>> + * component. Job initialization is split into two parts,
>> drm_sched_job_init()
>> + * and drm_sched_job_arm(). It's important to note that after arming
>> a job
>> + * drivers must follow the DMA-fence rules and can't easily allocate
>> memory
>> + * or takes locks under which memory is allocated.
>> + */
>> +
>> +/**
>> + * DOC: Entity Object
>> + *
>> + * The entity object which is a container for jobs which should
>> execute
>> + * sequentially. Drivers should create an entity for each individual
>> context
>> + * they maintain for command submissions which can run in parallel.
>> + *
>> + * The lifetime of the entity should *not* exceed the lifetime of
>> the
>> + * userspace process it was created for and drivers should call the
>> + * drm_sched_entity_flush() function from their
>> file_operations.flush
>> + * callback. So it's possible that an entity object is not alive any
>> + * more while jobs from it are still running on the hardware.
>> + *
>> + * Background is that for compatibility reasons with existing
>> + * userspace all results of a command submission should become
>> visible
>> + * externally even after after a process exits. This is normal POSIX
>> behavior
>> + * for I/O operations.
>> + *
>> + * The problem with this approach is that GPU submissions contain
>> executable
>> + * shaders enabling processes to evade their termination by
>> offloading work to
>> + * the GPU. So when a process is terminated with a SIGKILL the
>> entity object
>> + * makes sure that jobs are freed without running them while still
>> maintaining
>> + * correct sequential order for signaling fences.
>> + */
>> +
>> +/**
>> + * DOC: Hardware Fence Object
>> + *
>> + * The hardware fence object is a DMA-fence provided by the driver
>> as result of
>> + * running jobs. Drivers need to make sure that the normal DMA-fence
>> semantics
>> + * are followed for this object. It's important to note that the
>> memory for
>> + * this object can *not* be allocated in the run_job callback since
>> that would
>> + * violate the requirements for the DMA-fence implementation. The
>> scheduler
>> + * maintains a timeout handler which triggers if this fence doesn't
>> signal in
>> + * a configurable time frame.
>> + *
>> + * The lifetime of this object follows DMA-fence ref-counting rules,
>> the
>> + * scheduler takes ownership of the reference returned by the driver
>> and drops
>> + * it when it's not needed any more.
>> + */
>> +
>> +/**
>> + * DOC: Scheduler Fence Object
>> + *
>> + * The scheduler fence object which encapsulates the whole time from
>> pushing
>> + * the job into the scheduler until the hardware has finished
>> processing it.
>> + * This is internally managed by the scheduler, but drivers can grab
>> additional
>> + * reference to it after arming a job. The implementation provides
>> DMA-fence
>> + * interfaces for signaling both scheduling of a command submission
>> as well as
>> + * finishing of processing.
>> + *
>> + * The lifetime of this object also follows normal DMA-fence ref-
>> counting
>> + * rules. The finished fence is the one normally exposed outside of
>> the
>> + * scheduler, but the driver can grab references to both the
>> scheduled as well
>> + * as the finished fence when needed for pipe-lining optimizations.
>> + */
>> +
>> +/**
>> + * DOC: Scheduler and Run Queue Objects
>> + *
>> + * The scheduler object itself does the actual work of selecting a
>> job and
>> + * pushing it to the hardware. Both FIFO and RR selection algorithm
>> are
>> + * supported, but FIFO is preferred for many use cases.
>> + *
>> + * The lifetime of the scheduler is managed by the driver using it.
>> Before
>> + * destroying the scheduler the driver must ensure that all hardware
>> processing
>> + * involving this scheduler object has finished by calling for
>> example
>> + * disable_irq(). It is *not* sufficient to wait for the hardware
>> fence here
>> + * since this doesn't guarantee that all callback processing has
>> finished.
>> + *
>> + * The run queue object is a container of entities for a certain
>> priority
>> + * level. This object is internally managed by the scheduler and
>> drivers
>> + * shouldn't touch them directly. The lifetime of run queues are
>> bound to the
>> + * schedulers lifetime.
>>    */
>>   
>>   /**
>> @@ -72,6 +166,42 @@
>>    * limit.
>>    */
>>   
>> +/**
>> + * DOC: Error and Timeout handling
>> + *
>> + * Errors schould be signaled by using dma_fence_set_error() on the
>> hardware
>> + * fence object before signaling it. Errors are then bubbled up from
>> the
>> + * hardware fence to the scheduler fence.
>> + *
>> + * The entity allows querying errors on the last run submission
>> using the
>> + * drm_sched_entity_error() function which can be used to cancel
>> queued
>> + * submissions in the run_job callback as well as preventing pushing
>> further
>> + * ones into the entity in the drivers submission function.
>> + *
>> + * When the hardware fence fails to signal in a configurable amount
>> of time the
>> + * timedout_job callback is issued. The driver should then follow
>> the procedure
>> + * described on the &struct drm_sched_backend_ops.timedout_job
>> callback (TODO:
>> + * The timeout handler should probably switch to using the hardware
>> fence as
>> + * parameter instead of the job. Otherwise the handling will always
>> race
>> + * between timing out and signaling the fence).
>> + *
>> + * The scheduler also used to provided functionality for re-
>> submitting jobs
>> + * with replacing the hardware fence during reset handling. This
>> functionality
>> + * is now marked as deprecated. This has proven to be fundamentally
>> racy and
>> + * not compatible with DMA-fence rules and shouldn't be used in any
>> new code.
>> + *
>> + * Additional there is the drm_sched_increase_karma() function which
>> tries to
>> + * find the entity which submitted a job and increases it's 'karma'
>> + * atomic variable to prevent re-submitting jobs from this entity.
>> This has
>> + * quite some overhead and re-submitting jobs is now marked as
>> deprecated. So
>> + * using this function is rather discouraged.
>> + *
>> + * Drivers can still re-create the GPU state should it be lost
>> during timeout
>> + * handling when they can guarantee that forward progress is made
>> and this
>> + * doesn't cause another timeout. But this is strongly hardware
>> specific and
>> + * out of the scope of the general GPU scheduler.
>> + */
>> +
>>   #include <linux/wait.h>
>>   #include <linux/sched.h>
>>   #include <linux/completion.h>