From mcanal at igalia.com Sat May 3 20:59:51 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Sat, 03 May 2025 17:59:51 -0300 Subject: [PATCH 0/8] drm/sched: Allow drivers to skip the reset with DRM_GPU_SCHED_STAT_RUNNING Message-ID: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> When the DRM scheduler times out, it's possible that the GPU isn't hung; instead, a job may still be running, and there may be no valid reason to reset the hardware. This can occur in two situations: 1. The GPU exposes some mechanism that ensures the GPU is still making progress. By checking this mechanism, we can safely skip the reset, rearm the timeout, and allow the job to continue running until completion. This is the case for v3d and Etnaviv. 2. TDR has fired before the IRQ that signals the fence. Consequently, the job actually finishes, but it triggers a timeout before signaling the completion fence. These two scenarios are problematic because we remove the job from the `sched->pending_list` before calling `sched->ops->timedout_job()`. This means that when the job finally signals completion (e.g. in the IRQ handler), the scheduler won't call `sched->ops->free_job()`. As a result, the job and its resources won't be freed, leading to a memory leak. For v3d specifically, we have observed that these memory leaks can be significant in certain scenarios, as reported by users in [1][2]. To address this situation, I submitted a patch similar to commit 704d3d60fec4 ("drm/etnaviv: don't block scheduler when GPU is still active") for v3d [3]. This patch has already landed in drm-misc-fixes and successfully resolved the users' issues. However, as I mentioned in [3], exposing the scheduler's internals within the drivers isn't ideal and I believe this specific situation can be addressed within the DRM scheduler framework. This series aims to resolve this issue by adding a new DRM sched status that allows a driver to skip the reset. This new status will indicate that the job should be reinserted into the pending list, and the driver will still signal its completion. The series can be divided into three parts: * Patch 1: Implementation of the new status in the DRM scheduler. * Patches 2-4: Some fixes to the DRM scheduler KUnit tests and the addition of a test for the new status. * Patches 5-8: Usage the new status in four different drivers. [1] https://gitlab.freedesktop.org/mesa/mesa/-/issues/12227 [2] https://github.com/raspberrypi/linux/issues/6817 [3] https://lore.kernel.org/dri-devel/20250430210643.57924-1-mcanal at igalia.com/T/ Best Regards, - Ma?ra --- Ma?ra Canal (8): drm/sched: Allow drivers to skip the reset and keep on running drm/sched: Always free the job after the timeout drm/sched: Reduce scheduler's timeout for timeout tests drm/sched: Add new test for DRM_GPU_SCHED_STAT_RUNNING drm/v3d: Use DRM_GPU_SCHED_STAT_RUNNING to skip the reset drm/etnaviv: Use DRM_GPU_SCHED_STAT_RUNNING to skip the reset drm/xe: Use DRM_GPU_SCHED_STAT_RUNNING to skip the reset drm/panfrost: Use DRM_GPU_SCHED_STAT_RUNNING to skip the reset drivers/gpu/drm/etnaviv/etnaviv_sched.c | 12 ++--- drivers/gpu/drm/panfrost/panfrost_job.c | 8 ++-- drivers/gpu/drm/scheduler/sched_main.c | 14 ++++++ drivers/gpu/drm/scheduler/tests/mock_scheduler.c | 13 ++++++ drivers/gpu/drm/scheduler/tests/tests_basic.c | 57 ++++++++++++++++++++++-- drivers/gpu/drm/v3d/v3d_sched.c | 4 +- drivers/gpu/drm/xe/xe_guc_submit.c | 8 +--- include/drm/gpu_scheduler.h | 2 + 8 files changed, 94 insertions(+), 24 deletions(-) --- base-commit: 760e296124ef3b6e14cd1d940f2a01c5ed7c0dac change-id: 20250502-sched-skip-reset-bf7c163233da From mcanal at igalia.com Sat May 3 20:59:52 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Sat, 03 May 2025 17:59:52 -0300 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> Message-ID: <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> When the DRM scheduler times out, it's possible that the GPU isn't hung; instead, a job may still be running, and there may be no valid reason to reset the hardware. This can occur in two situations: 1. The GPU exposes some mechanism that ensures the GPU is still making progress. By checking this mechanism, we can safely skip the reset, rearm the timeout, and allow the job to continue running until completion. This is the case for v3d and Etnaviv. 2. TDR has fired before the IRQ that signals the fence. Consequently, the job actually finishes, but it triggers a timeout before signaling the completion fence. These two scenarios are problematic because we remove the job from the `sched->pending_list` before calling `sched->ops->timedout_job()`. This means that when the job finally signals completion (e.g. in the IRQ handler), the scheduler won't call `sched->ops->free_job()`. As a result, the job and its resources won't be freed, leading to a memory leak. To resolve this issue, we create a new `drm_gpu_sched_stat` that allows a driver to skip the reset. This new status will indicate that the job should be reinserted into the pending list, and the driver will still signal its completion. Signed-off-by: Ma?ra Canal --- drivers/gpu/drm/scheduler/sched_main.c | 14 ++++++++++++++ include/drm/gpu_scheduler.h | 2 ++ 2 files changed, 16 insertions(+) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 829579c41c6b5d8b2abce5ad373c7017469b7680..68ca827d77e32187a034309f881135dbc639a9b4 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -568,6 +568,17 @@ static void drm_sched_job_timedout(struct work_struct *work) job->sched->ops->free_job(job); sched->free_guilty = false; } + + /* + * If the driver indicated that the GPU is still running and wants to skip + * the reset, reinsert the job back into the pending list and realarm the + * timeout. + */ + if (status == DRM_GPU_SCHED_STAT_RUNNING) { + spin_lock(&sched->job_list_lock); + list_add(&job->list, &sched->pending_list); + spin_unlock(&sched->job_list_lock); + } } else { spin_unlock(&sched->job_list_lock); } @@ -590,6 +601,9 @@ static void drm_sched_job_timedout(struct work_struct *work) * This function is typically used for reset recovery (see the docu of * drm_sched_backend_ops.timedout_job() for details). Do not call it for * scheduler teardown, i.e., before calling drm_sched_fini(). + * + * As it's used for reset recovery, drm_sched_stop() shouldn't be called + * if the scheduler skipped the timeout (DRM_SCHED_STAT_RUNNING). */ void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad) { diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 1a7e377d4cbb4fc12ed93c548b236970217945e8..fe9043b6d43141bee831b5fc16b927202a507d51 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -389,11 +389,13 @@ struct drm_sched_job { * @DRM_GPU_SCHED_STAT_NONE: Reserved. Do not use. * @DRM_GPU_SCHED_STAT_NOMINAL: Operation succeeded. * @DRM_GPU_SCHED_STAT_ENODEV: Error: Device is not available anymore. + * @DRM_GPU_SCHED_STAT_RUNNING: GPU is still running, so skip the reset. */ enum drm_gpu_sched_stat { DRM_GPU_SCHED_STAT_NONE, DRM_GPU_SCHED_STAT_NOMINAL, DRM_GPU_SCHED_STAT_ENODEV, + DRM_GPU_SCHED_STAT_RUNNING, }; /** -- 2.49.0 From mcanal at igalia.com Sat May 3 20:59:53 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Sat, 03 May 2025 17:59:53 -0300 Subject: [PATCH 2/8] drm/sched: Always free the job after the timeout In-Reply-To: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> Message-ID: <20250503-sched-skip-reset-v1-2-ed0d6701a3fe@igalia.com> Currently, if we add the assertions presented in this commit to the mock scheduler, we will see the following output: [15:47:08] ============== [PASSED] drm_sched_basic_tests ============== [15:47:08] ======== drm_sched_basic_timeout_tests (1 subtest) ========= [15:47:08] # drm_sched_basic_timeout: ASSERTION FAILED at drivers/gpu/drm/scheduler/tests/tests_basic.c:246 [15:47:08] Expected list_empty(&sched->job_list) to be true, but is false [15:47:08] [FAILED] drm_sched_basic_timeout [15:47:08] # module: drm_sched_tests This occurs because `mock_sched_timedout_job()` doesn't properly handle the hang. From the DRM sched documentation, `drm_sched_stop()` and `drm_sched_start()` are typically used for reset recovery. If these functions are not used, the offending job won't be freed and should be freed by the caller. Currently, the mock scheduler doesn't use the functions provided by the API, nor does it handle the freeing of the job. As a result, the job isn't removed from the job list. This commit mocks a GPU reset by stopping the scheduler affected by the reset, waiting a couple of microseconds to mimic a hardware reset, and then restart the affected scheduler. Signed-off-by: Ma?ra Canal --- drivers/gpu/drm/scheduler/tests/mock_scheduler.c | 10 ++++++++++ drivers/gpu/drm/scheduler/tests/tests_basic.c | 3 +++ 2 files changed, 13 insertions(+) diff --git a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c index f999c8859cf7adb8f06fc8a37969656dd3249fa7..e9af202d84bd55ea5cc048215e39f5407bc84458 100644 --- a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c +++ b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c @@ -1,6 +1,8 @@ // SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2025 Valve Corporation */ +#include + #include "sched_tests.h" /* @@ -203,10 +205,18 @@ static struct dma_fence *mock_sched_run_job(struct drm_sched_job *sched_job) static enum drm_gpu_sched_stat mock_sched_timedout_job(struct drm_sched_job *sched_job) { + struct drm_mock_scheduler *sched = + drm_sched_to_mock_sched(sched_job->sched); struct drm_mock_sched_job *job = drm_sched_job_to_mock_job(sched_job); job->flags |= DRM_MOCK_SCHED_JOB_TIMEDOUT; + drm_sched_stop(&sched->base, &job->base); + + usleep_range(200, 500); + + drm_sched_start(&sched->base, 0); + return DRM_GPU_SCHED_STAT_NOMINAL; } diff --git a/drivers/gpu/drm/scheduler/tests/tests_basic.c b/drivers/gpu/drm/scheduler/tests/tests_basic.c index 7230057e0594c6246f02608f07fcb1f8d738ac75..8f960f0fd31d0af7873f410ceba2d636f58a5474 100644 --- a/drivers/gpu/drm/scheduler/tests/tests_basic.c +++ b/drivers/gpu/drm/scheduler/tests/tests_basic.c @@ -241,6 +241,9 @@ static void drm_sched_basic_timeout(struct kunit *test) job->flags & DRM_MOCK_SCHED_JOB_TIMEDOUT, DRM_MOCK_SCHED_JOB_TIMEDOUT); + KUNIT_ASSERT_TRUE(test, list_empty(&sched->job_list)); + KUNIT_ASSERT_TRUE(test, list_empty(&sched->done_list)); + drm_mock_sched_entity_free(entity); } -- 2.49.0 From mcanal at igalia.com Sat May 3 20:59:54 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Sat, 03 May 2025 17:59:54 -0300 Subject: [PATCH 3/8] drm/sched: Reduce scheduler's timeout for timeout tests In-Reply-To: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> Message-ID: <20250503-sched-skip-reset-v1-3-ed0d6701a3fe@igalia.com> As more KUnit tests are introduced to evaluate the basic capabilities of the `timedout_job()` hook, the test suite will continue to increase in duration. To reduce the overall running time of the test suite, decrease the scheduler's timeout for the timeout tests. Before this commit: [15:42:26] Elapsed time: 15.637s total, 0.002s configuring, 10.387s building, 5.229s running After this commit: [15:45:26] Elapsed time: 9.263s total, 0.002s configuring, 5.168s building, 4.037s running Signed-off-by: Ma?ra Canal --- drivers/gpu/drm/scheduler/tests/tests_basic.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/drivers/gpu/drm/scheduler/tests/tests_basic.c b/drivers/gpu/drm/scheduler/tests/tests_basic.c index 8f960f0fd31d0af7873f410ceba2d636f58a5474..00c691cb3c306f609684f554f17fcb54ba74cb95 100644 --- a/drivers/gpu/drm/scheduler/tests/tests_basic.c +++ b/drivers/gpu/drm/scheduler/tests/tests_basic.c @@ -5,6 +5,8 @@ #include "sched_tests.h" +#define MOCK_TIMEOUT (HZ / 5) + /* * DRM scheduler basic tests should check the basic functional correctness of * the scheduler, including some very light smoke testing. More targeted tests, @@ -28,7 +30,7 @@ static void drm_sched_basic_exit(struct kunit *test) static int drm_sched_timeout_init(struct kunit *test) { - test->priv = drm_mock_sched_new(test, HZ); + test->priv = drm_mock_sched_new(test, MOCK_TIMEOUT); return 0; } @@ -224,17 +226,17 @@ static void drm_sched_basic_timeout(struct kunit *test) drm_mock_sched_job_submit(job); - done = drm_mock_sched_job_wait_scheduled(job, HZ); + done = drm_mock_sched_job_wait_scheduled(job, MOCK_TIMEOUT); KUNIT_ASSERT_TRUE(test, done); - done = drm_mock_sched_job_wait_finished(job, HZ / 2); + done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT / 2); KUNIT_ASSERT_FALSE(test, done); KUNIT_ASSERT_EQ(test, job->flags & DRM_MOCK_SCHED_JOB_TIMEDOUT, 0); - done = drm_mock_sched_job_wait_finished(job, HZ); + done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT); KUNIT_ASSERT_FALSE(test, done); KUNIT_ASSERT_EQ(test, -- 2.49.0 From mcanal at igalia.com Sat May 3 20:59:55 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Sat, 03 May 2025 17:59:55 -0300 Subject: [PATCH 4/8] drm/sched: Add new test for DRM_GPU_SCHED_STAT_RUNNING In-Reply-To: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> Message-ID: <20250503-sched-skip-reset-v1-4-ed0d6701a3fe@igalia.com> Add a test to submit a single job against a scheduler with the timeout configured and verify that if the job is still running, the timeout handler will skip the reset and allow the job to complete. Signed-off-by: Ma?ra Canal --- drivers/gpu/drm/scheduler/tests/mock_scheduler.c | 3 ++ drivers/gpu/drm/scheduler/tests/tests_basic.c | 44 ++++++++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c index e9af202d84bd55ea5cc048215e39f5407bc84458..9d594cb5bf567be25e018ddbcd28b70a7e994260 100644 --- a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c +++ b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c @@ -211,6 +211,9 @@ mock_sched_timedout_job(struct drm_sched_job *sched_job) job->flags |= DRM_MOCK_SCHED_JOB_TIMEDOUT; + if (job->finish_at && ktime_before(ktime_get(), job->finish_at)) + return DRM_GPU_SCHED_STAT_RUNNING; + drm_sched_stop(&sched->base, &job->base); usleep_range(200, 500); diff --git a/drivers/gpu/drm/scheduler/tests/tests_basic.c b/drivers/gpu/drm/scheduler/tests/tests_basic.c index 00c691cb3c306f609684f554f17fcb54ba74cb95..669a211b216ee298544ac237abb866077d856586 100644 --- a/drivers/gpu/drm/scheduler/tests/tests_basic.c +++ b/drivers/gpu/drm/scheduler/tests/tests_basic.c @@ -249,8 +249,52 @@ static void drm_sched_basic_timeout(struct kunit *test) drm_mock_sched_entity_free(entity); } +static void drm_sched_skip_reset(struct kunit *test) +{ + struct drm_mock_scheduler *sched = test->priv; + struct drm_mock_sched_entity *entity; + struct drm_mock_sched_job *job; + bool done; + + /* + * Submit a single job against a scheduler with the timeout configured + * and verify that if the job is still running, the timeout handler + * will skip the reset and allow the job to complete. + */ + + entity = drm_mock_sched_entity_new(test, + DRM_SCHED_PRIORITY_NORMAL, + sched); + job = drm_mock_sched_job_new(test, entity); + + drm_mock_sched_job_set_duration_us(job, jiffies_to_usecs(2 * MOCK_TIMEOUT)); + drm_mock_sched_job_submit(job); + + done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT); + KUNIT_ASSERT_FALSE(test, done); + + KUNIT_ASSERT_EQ(test, + job->flags & DRM_MOCK_SCHED_JOB_TIMEDOUT, + DRM_MOCK_SCHED_JOB_TIMEDOUT); + + KUNIT_ASSERT_FALSE(test, list_empty(&sched->job_list)); + + done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT); + KUNIT_ASSERT_TRUE(test, done); + + KUNIT_ASSERT_EQ(test, + job->flags & DRM_MOCK_SCHED_JOB_TIMEDOUT, + DRM_MOCK_SCHED_JOB_TIMEDOUT); + + KUNIT_ASSERT_TRUE(test, list_empty(&sched->job_list)); + KUNIT_ASSERT_TRUE(test, list_empty(&sched->done_list)); + + drm_mock_sched_entity_free(entity); +} + static struct kunit_case drm_sched_timeout_tests[] = { KUNIT_CASE(drm_sched_basic_timeout), + KUNIT_CASE(drm_sched_skip_reset), {} }; -- 2.49.0 From mcanal at igalia.com Sat May 3 20:59:56 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Sat, 03 May 2025 17:59:56 -0300 Subject: [PATCH 5/8] drm/v3d: Use DRM_GPU_SCHED_STAT_RUNNING to skip the reset In-Reply-To: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> Message-ID: <20250503-sched-skip-reset-v1-5-ed0d6701a3fe@igalia.com> When a CL/CSD job times out, we check if the GPU has made any progress since the last timeout. If so, instead of resetting the hardware, we skip the reset and allow the timer to be rearmed. This gives long-running jobs a chance to complete. Use the DRM_GPU_SCHED_STAT_RUNNING status to skip the reset and rearm the timer. Signed-off-by: Ma?ra Canal --- drivers/gpu/drm/v3d/v3d_sched.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/v3d/v3d_sched.c b/drivers/gpu/drm/v3d/v3d_sched.c index b3be08b0ca9188564f9fb6aa32694940a5fadc9d..51770b6686d0befffa3e87c290bbdc1a12a19ad5 100644 --- a/drivers/gpu/drm/v3d/v3d_sched.c +++ b/drivers/gpu/drm/v3d/v3d_sched.c @@ -751,7 +751,7 @@ v3d_cl_job_timedout(struct drm_sched_job *sched_job, enum v3d_queue q, if (*timedout_ctca != ctca || *timedout_ctra != ctra) { *timedout_ctca = ctca; *timedout_ctra = ctra; - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RUNNING; } return v3d_gpu_reset_for_timeout(v3d, sched_job); @@ -795,7 +795,7 @@ v3d_csd_job_timedout(struct drm_sched_job *sched_job) */ if (job->timedout_batches != batches) { job->timedout_batches = batches; - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RUNNING; } return v3d_gpu_reset_for_timeout(v3d, sched_job); -- 2.49.0 From mcanal at igalia.com Sat May 3 20:59:57 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Sat, 03 May 2025 17:59:57 -0300 Subject: [PATCH 6/8] drm/etnaviv: Use DRM_GPU_SCHED_STAT_RUNNING to skip the reset In-Reply-To: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> Message-ID: <20250503-sched-skip-reset-v1-6-ed0d6701a3fe@igalia.com> Etnaviv can skip a hardware reset in two situations: 1. TDR has fired before the IRQ and the timeout is spurious. 2. The GPU is still making progress on the front-end and we can give the job a chance to complete. Instead of relying on the scheduler internals, use the DRM_GPU_SCHED_STAT_RUNNING status to skip the reset and rearm the timer. Signed-off-by: Ma?ra Canal --- drivers/gpu/drm/etnaviv/etnaviv_sched.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/drivers/gpu/drm/etnaviv/etnaviv_sched.c b/drivers/gpu/drm/etnaviv/etnaviv_sched.c index 76a3a3e517d8d9f654fb6b9e98e72910795cfc7a..b87ffdb4136aebade736d78b3677de2f21d52ebc 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_sched.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_sched.c @@ -40,11 +40,11 @@ static enum drm_gpu_sched_stat etnaviv_sched_timedout_job(struct drm_sched_job int change; /* - * If the GPU managed to complete this jobs fence, the timout is - * spurious. Bail out. + * If the GPU managed to complete this jobs fence, TDR has fired before + * IRQ and the timeout is spurious. Bail out. */ if (dma_fence_is_signaled(submit->out_fence)) - goto out_no_timeout; + return DRM_GPU_SCHED_STAT_RUNNING; /* * If the GPU is still making forward progress on the front-end (which @@ -70,7 +70,7 @@ static enum drm_gpu_sched_stat etnaviv_sched_timedout_job(struct drm_sched_job gpu->hangcheck_dma_addr = dma_addr; gpu->hangcheck_primid = primid; gpu->hangcheck_fence = gpu->completed_fence; - goto out_no_timeout; + return DRM_GPU_SCHED_STAT_RUNNING; } /* block scheduler */ @@ -87,10 +87,6 @@ static enum drm_gpu_sched_stat etnaviv_sched_timedout_job(struct drm_sched_job drm_sched_start(&gpu->sched, 0); return DRM_GPU_SCHED_STAT_NOMINAL; - -out_no_timeout: - list_add(&sched_job->list, &sched_job->sched->pending_list); - return DRM_GPU_SCHED_STAT_NOMINAL; } static void etnaviv_sched_free_job(struct drm_sched_job *sched_job) -- 2.49.0 From mcanal at igalia.com Sat May 3 20:59:58 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Sat, 03 May 2025 17:59:58 -0300 Subject: [PATCH 7/8] drm/xe: Use DRM_GPU_SCHED_STAT_RUNNING to skip the reset In-Reply-To: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> Message-ID: <20250503-sched-skip-reset-v1-7-ed0d6701a3fe@igalia.com> Xe can skip the reset if TDR has fired before the free job worker. Instead of using the scheduler internals to add the job to the pending list, use the DRM_GPU_SCHED_STAT_RUNNING status to skip the reset and rearm the timer. Note that there is no need to restart submission if it hasn't been stopped. Signed-off-by: Ma?ra Canal --- drivers/gpu/drm/xe/xe_guc_submit.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c index 31bc2022bfc2d80f0ef54726dfeb8d7f8e6b32c8..4c40d3921d4a5e190d3413736a68c6e7295223dd 100644 --- a/drivers/gpu/drm/xe/xe_guc_submit.c +++ b/drivers/gpu/drm/xe/xe_guc_submit.c @@ -1058,12 +1058,8 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job) * list so job can be freed and kick scheduler ensuring free job is not * lost. */ - if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &job->fence->flags)) { - xe_sched_add_pending_job(sched, job); - xe_sched_submission_start(sched); - - return DRM_GPU_SCHED_STAT_NOMINAL; - } + if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &job->fence->flags)) + return DRM_GPU_SCHED_STAT_RUNNING; /* Kill the run_job entry point */ xe_sched_submission_stop(sched); -- 2.49.0 From mcanal at igalia.com Sat May 3 20:59:59 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Sat, 03 May 2025 17:59:59 -0300 Subject: [PATCH 8/8] drm/panfrost: Use DRM_GPU_SCHED_STAT_RUNNING to skip the reset In-Reply-To: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> Message-ID: <20250503-sched-skip-reset-v1-8-ed0d6701a3fe@igalia.com> Panfrost can skip the reset if TDR has fired before the IRQ handler. Currently, since Panfrost doesn't take any action on these scenarios, the job is being leaked, considering that `free_job()` won't be called. To avoid such leaks, use the DRM_GPU_SCHED_STAT_RUNNING status to skip the reset and rearm the timer. Signed-off-by: Ma?ra Canal --- drivers/gpu/drm/panfrost/panfrost_job.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c index 5657106c2f7d0a0ca6162850767f58f3200cce13..2948d5c02115544a0e0babffd850f1506152849d 100644 --- a/drivers/gpu/drm/panfrost/panfrost_job.c +++ b/drivers/gpu/drm/panfrost/panfrost_job.c @@ -751,11 +751,11 @@ static enum drm_gpu_sched_stat panfrost_job_timedout(struct drm_sched_job int js = panfrost_job_get_slot(job); /* - * If the GPU managed to complete this jobs fence, the timeout is - * spurious. Bail out. + * If the GPU managed to complete this jobs fence, TDR has fired before + * IRQ and the timeout is spurious. Bail out. */ if (dma_fence_is_signaled(job->done_fence)) - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RUNNING; /* * Panfrost IRQ handler may take a long time to process an interrupt @@ -770,7 +770,7 @@ static enum drm_gpu_sched_stat panfrost_job_timedout(struct drm_sched_job if (dma_fence_is_signaled(job->done_fence)) { dev_warn(pfdev->dev, "unexpectedly high interrupt latency\n"); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RUNNING; } dev_err(pfdev->dev, "gpu sched timeout, js=%d, config=0x%x, status=0x%x, head=0x%x, tail=0x%x, sched_job=%p", -- 2.49.0 From matthew.brost at intel.com Tue May 6 02:41:09 2025 From: matthew.brost at intel.com (Matthew Brost) Date: Mon, 5 May 2025 19:41:09 -0700 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> Message-ID: On Sat, May 03, 2025 at 05:59:52PM -0300, Ma?ra Canal wrote: > When the DRM scheduler times out, it's possible that the GPU isn't hung; > instead, a job may still be running, and there may be no valid reason to > reset the hardware. This can occur in two situations: > > 1. The GPU exposes some mechanism that ensures the GPU is still making > progress. By checking this mechanism, we can safely skip the reset, > rearm the timeout, and allow the job to continue running until > completion. This is the case for v3d and Etnaviv. > 2. TDR has fired before the IRQ that signals the fence. Consequently, > the job actually finishes, but it triggers a timeout before signaling > the completion fence. > We have both of these cases in Xe too. We implement the requeuing in Xe via driver side function - xe_sched_add_pending_job but this looks better and will make use of this. > These two scenarios are problematic because we remove the job from the > `sched->pending_list` before calling `sched->ops->timedout_job()`. This > means that when the job finally signals completion (e.g. in the IRQ > handler), the scheduler won't call `sched->ops->free_job()`. As a result, > the job and its resources won't be freed, leading to a memory leak. > > To resolve this issue, we create a new `drm_gpu_sched_stat` that allows a > driver to skip the reset. This new status will indicate that the job > should be reinserted into the pending list, and the driver will still > signal its completion. > > Signed-off-by: Ma?ra Canal Reviewed-by: Matthew Brost > --- > drivers/gpu/drm/scheduler/sched_main.c | 14 ++++++++++++++ > include/drm/gpu_scheduler.h | 2 ++ > 2 files changed, 16 insertions(+) > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c > index 829579c41c6b5d8b2abce5ad373c7017469b7680..68ca827d77e32187a034309f881135dbc639a9b4 100644 > --- a/drivers/gpu/drm/scheduler/sched_main.c > +++ b/drivers/gpu/drm/scheduler/sched_main.c > @@ -568,6 +568,17 @@ static void drm_sched_job_timedout(struct work_struct *work) > job->sched->ops->free_job(job); > sched->free_guilty = false; > } > + > + /* > + * If the driver indicated that the GPU is still running and wants to skip > + * the reset, reinsert the job back into the pending list and realarm the > + * timeout. > + */ > + if (status == DRM_GPU_SCHED_STAT_RUNNING) { > + spin_lock(&sched->job_list_lock); > + list_add(&job->list, &sched->pending_list); > + spin_unlock(&sched->job_list_lock); > + } > } else { > spin_unlock(&sched->job_list_lock); > } > @@ -590,6 +601,9 @@ static void drm_sched_job_timedout(struct work_struct *work) > * This function is typically used for reset recovery (see the docu of > * drm_sched_backend_ops.timedout_job() for details). Do not call it for > * scheduler teardown, i.e., before calling drm_sched_fini(). > + * > + * As it's used for reset recovery, drm_sched_stop() shouldn't be called > + * if the scheduler skipped the timeout (DRM_SCHED_STAT_RUNNING). > */ > void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad) > { > diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h > index 1a7e377d4cbb4fc12ed93c548b236970217945e8..fe9043b6d43141bee831b5fc16b927202a507d51 100644 > --- a/include/drm/gpu_scheduler.h > +++ b/include/drm/gpu_scheduler.h > @@ -389,11 +389,13 @@ struct drm_sched_job { > * @DRM_GPU_SCHED_STAT_NONE: Reserved. Do not use. > * @DRM_GPU_SCHED_STAT_NOMINAL: Operation succeeded. > * @DRM_GPU_SCHED_STAT_ENODEV: Error: Device is not available anymore. > + * @DRM_GPU_SCHED_STAT_RUNNING: GPU is still running, so skip the reset. > */ > enum drm_gpu_sched_stat { > DRM_GPU_SCHED_STAT_NONE, > DRM_GPU_SCHED_STAT_NOMINAL, > DRM_GPU_SCHED_STAT_ENODEV, > + DRM_GPU_SCHED_STAT_RUNNING, > }; > > /** > > -- > 2.49.0 > From matthew.brost at intel.com Tue May 6 04:25:09 2025 From: matthew.brost at intel.com (Matthew Brost) Date: Mon, 5 May 2025 21:25:09 -0700 Subject: [PATCH 7/8] drm/xe: Use DRM_GPU_SCHED_STAT_RUNNING to skip the reset In-Reply-To: <20250503-sched-skip-reset-v1-7-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-7-ed0d6701a3fe@igalia.com> Message-ID: On Sat, May 03, 2025 at 05:59:58PM -0300, Ma?ra Canal wrote: > Xe can skip the reset if TDR has fired before the free job worker. Instead > of using the scheduler internals to add the job to the pending list, use > the DRM_GPU_SCHED_STAT_RUNNING status to skip the reset and rearm the > timer. > > Note that there is no need to restart submission if it hasn't been > stopped. > > Signed-off-by: Ma?ra Canal > --- > drivers/gpu/drm/xe/xe_guc_submit.c | 8 ++------ > 1 file changed, 2 insertions(+), 6 deletions(-) > > diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c > index 31bc2022bfc2d80f0ef54726dfeb8d7f8e6b32c8..4c40d3921d4a5e190d3413736a68c6e7295223dd 100644 > --- a/drivers/gpu/drm/xe/xe_guc_submit.c > +++ b/drivers/gpu/drm/xe/xe_guc_submit.c > @@ -1058,12 +1058,8 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job) > * list so job can be freed and kick scheduler ensuring free job is not > * lost. > */ > - if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &job->fence->flags)) { > - xe_sched_add_pending_job(sched, job); > - xe_sched_submission_start(sched); > - > - return DRM_GPU_SCHED_STAT_NOMINAL; > - } > + if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &job->fence->flags)) > + return DRM_GPU_SCHED_STAT_RUNNING; There are a couple of other calls to xe_sched_add_pending_job in this function which I can believe can be dropped in favor of return DRM_GPU_SCHED_STAT_RUNNING too. Matt > > /* Kill the run_job entry point */ > xe_sched_submission_stop(sched); > > -- > 2.49.0 > From tvrtko.ursulin at igalia.com Tue May 6 11:32:02 2025 From: tvrtko.ursulin at igalia.com (Tvrtko Ursulin) Date: Tue, 6 May 2025 12:32:02 +0100 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> Message-ID: On 03/05/2025 21:59, Ma?ra Canal wrote: > When the DRM scheduler times out, it's possible that the GPU isn't hung; > instead, a job may still be running, and there may be no valid reason to > reset the hardware. This can occur in two situations: > > 1. The GPU exposes some mechanism that ensures the GPU is still making > progress. By checking this mechanism, we can safely skip the reset, > rearm the timeout, and allow the job to continue running until > completion. This is the case for v3d and Etnaviv. > 2. TDR has fired before the IRQ that signals the fence. Consequently, > the job actually finishes, but it triggers a timeout before signaling > the completion fence. > > These two scenarios are problematic because we remove the job from the > `sched->pending_list` before calling `sched->ops->timedout_job()`. This > means that when the job finally signals completion (e.g. in the IRQ > handler), the scheduler won't call `sched->ops->free_job()`. As a result, > the job and its resources won't be freed, leading to a memory leak. > > To resolve this issue, we create a new `drm_gpu_sched_stat` that allows a > driver to skip the reset. This new status will indicate that the job > should be reinserted into the pending list, and the driver will still > signal its completion. Since this is de facto what drivers do today I agree it makes sense to formalise handling for it in the scheduler itself. Acked-by: Tvrtko Ursulin Some minor comments below. > Signed-off-by: Ma?ra Canal > --- > drivers/gpu/drm/scheduler/sched_main.c | 14 ++++++++++++++ > include/drm/gpu_scheduler.h | 2 ++ > 2 files changed, 16 insertions(+) > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c > index 829579c41c6b5d8b2abce5ad373c7017469b7680..68ca827d77e32187a034309f881135dbc639a9b4 100644 > --- a/drivers/gpu/drm/scheduler/sched_main.c > +++ b/drivers/gpu/drm/scheduler/sched_main.c > @@ -568,6 +568,17 @@ static void drm_sched_job_timedout(struct work_struct *work) > job->sched->ops->free_job(job); > sched->free_guilty = false; > } > + > + /* > + * If the driver indicated that the GPU is still running and wants to skip > + * the reset, reinsert the job back into the pending list and realarm the re-arm > + * timeout. > + */ > + if (status == DRM_GPU_SCHED_STAT_RUNNING) { > + spin_lock(&sched->job_list_lock); > + list_add(&job->list, &sched->pending_list); > + spin_unlock(&sched->job_list_lock); > + } > } else { > spin_unlock(&sched->job_list_lock); > } > @@ -590,6 +601,9 @@ static void drm_sched_job_timedout(struct work_struct *work) > * This function is typically used for reset recovery (see the docu of > * drm_sched_backend_ops.timedout_job() for details). Do not call it for > * scheduler teardown, i.e., before calling drm_sched_fini(). > + * > + * As it's used for reset recovery, drm_sched_stop() shouldn't be called > + * if the scheduler skipped the timeout (DRM_SCHED_STAT_RUNNING). s/scheduler/driver/ ? > */ > void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad) > { > diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h > index 1a7e377d4cbb4fc12ed93c548b236970217945e8..fe9043b6d43141bee831b5fc16b927202a507d51 100644 > --- a/include/drm/gpu_scheduler.h > +++ b/include/drm/gpu_scheduler.h > @@ -389,11 +389,13 @@ struct drm_sched_job { > * @DRM_GPU_SCHED_STAT_NONE: Reserved. Do not use. > * @DRM_GPU_SCHED_STAT_NOMINAL: Operation succeeded. > * @DRM_GPU_SCHED_STAT_ENODEV: Error: Device is not available anymore. > + * @DRM_GPU_SCHED_STAT_RUNNING: GPU is still running, so skip the reset. s/GPU/job/ ? > */ > enum drm_gpu_sched_stat { > DRM_GPU_SCHED_STAT_NONE, > DRM_GPU_SCHED_STAT_NOMINAL, > DRM_GPU_SCHED_STAT_ENODEV, > + DRM_GPU_SCHED_STAT_RUNNING, I am wondering if we could make it more obvious what is the difference between "nominal" and "running" and from whose point of view should those statuses be considered. So far we have "nominal" which means scheduler/hardware is working fine but the job may or may have not been cancelled. With "running" we kind of split it into two sub-statuses and it would be nice for that to be intuitively visible from the naming. But I struggle to suggest an elegant name while preserving nominal as is. Thinking out loud here - perhaps that is pointing towards an alternative that instead of a new status, a new helper to re-insert the single job (like drm_sched_resubmit_job(sched, job)) would fit better? Although it would be more churn. Regards, Tvrtko > }; > > /** > From tvrtko.ursulin at igalia.com Tue May 6 11:49:20 2025 From: tvrtko.ursulin at igalia.com (Tvrtko Ursulin) Date: Tue, 6 May 2025 12:49:20 +0100 Subject: [PATCH 2/8] drm/sched: Always free the job after the timeout In-Reply-To: <20250503-sched-skip-reset-v1-2-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-2-ed0d6701a3fe@igalia.com> Message-ID: <3fe178ec-9c16-4abc-b302-64f0077d8af4@igalia.com> On 03/05/2025 21:59, Ma?ra Canal wrote: > Currently, if we add the assertions presented in this commit to the mock > scheduler, we will see the following output: > > [15:47:08] ============== [PASSED] drm_sched_basic_tests ============== > [15:47:08] ======== drm_sched_basic_timeout_tests (1 subtest) ========= > [15:47:08] # drm_sched_basic_timeout: ASSERTION FAILED at drivers/gpu/drm/scheduler/tests/tests_basic.c:246 > [15:47:08] Expected list_empty(&sched->job_list) to be true, but is false > [15:47:08] [FAILED] drm_sched_basic_timeout > [15:47:08] # module: drm_sched_tests > > This occurs because `mock_sched_timedout_job()` doesn't properly handle > the hang. From the DRM sched documentation, `drm_sched_stop()` and > `drm_sched_start()` are typically used for reset recovery. If these > functions are not used, the offending job won't be freed and should be > freed by the caller. > > Currently, the mock scheduler doesn't use the functions provided by the > API, nor does it handle the freeing of the job. As a result, the job isn't > removed from the job list. For the record the job does gets freed via the kunit managed allocation. It was a design choice for this test to be a *strict* unit test which tests only a _single_ thing. And that is that the timedout_job() hook gets called. As such the hook was implemented to satisfy that single requirement only. But I also do not oppose making it test multiple things in one test per se. > This commit mocks a GPU reset by stopping the scheduler affected by the > reset, waiting a couple of microseconds to mimic a hardware reset, and > then restart the affected scheduler. > > Signed-off-by: Ma?ra Canal > --- > drivers/gpu/drm/scheduler/tests/mock_scheduler.c | 10 ++++++++++ > drivers/gpu/drm/scheduler/tests/tests_basic.c | 3 +++ > 2 files changed, 13 insertions(+) > > diff --git a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c > index f999c8859cf7adb8f06fc8a37969656dd3249fa7..e9af202d84bd55ea5cc048215e39f5407bc84458 100644 > --- a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c > +++ b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c > @@ -1,6 +1,8 @@ > // SPDX-License-Identifier: GPL-2.0 > /* Copyright (c) 2025 Valve Corporation */ > > +#include > + > #include "sched_tests.h" > > /* > @@ -203,10 +205,18 @@ static struct dma_fence *mock_sched_run_job(struct drm_sched_job *sched_job) > static enum drm_gpu_sched_stat > mock_sched_timedout_job(struct drm_sched_job *sched_job) > { > + struct drm_mock_scheduler *sched = > + drm_sched_to_mock_sched(sched_job->sched); > struct drm_mock_sched_job *job = drm_sched_job_to_mock_job(sched_job); > > job->flags |= DRM_MOCK_SCHED_JOB_TIMEDOUT; > > + drm_sched_stop(&sched->base, &job->base); > + > + usleep_range(200, 500); msleep(10) or something to make it seem less like the actual numbers are relevant? > +> + drm_sched_start(&sched->base, 0); > + > return DRM_GPU_SCHED_STAT_NOMINAL; > } > > diff --git a/drivers/gpu/drm/scheduler/tests/tests_basic.c b/drivers/gpu/drm/scheduler/tests/tests_basic.c > index 7230057e0594c6246f02608f07fcb1f8d738ac75..8f960f0fd31d0af7873f410ceba2d636f58a5474 100644 > --- a/drivers/gpu/drm/scheduler/tests/tests_basic.c > +++ b/drivers/gpu/drm/scheduler/tests/tests_basic.c > @@ -241,6 +241,9 @@ static void drm_sched_basic_timeout(struct kunit *test) > job->flags & DRM_MOCK_SCHED_JOB_TIMEDOUT, > DRM_MOCK_SCHED_JOB_TIMEDOUT); > > + KUNIT_ASSERT_TRUE(test, list_empty(&sched->job_list)); Hmm I think this assert could be racy because it appears to rely on the free worker to run and cleanup the "finished" job in the window between drm_mock_sched_job_wait_finished() (or drm_sched_start(), depends how you look at it) and here. Am I missing something? Regards, Tvrtko > + KUNIT_ASSERT_TRUE(test, list_empty(&sched->done_list)); > +> drm_mock_sched_entity_free(entity); > } > > From tvrtko.ursulin at igalia.com Tue May 6 12:03:05 2025 From: tvrtko.ursulin at igalia.com (Tvrtko Ursulin) Date: Tue, 6 May 2025 13:03:05 +0100 Subject: [PATCH 3/8] drm/sched: Reduce scheduler's timeout for timeout tests In-Reply-To: <20250503-sched-skip-reset-v1-3-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-3-ed0d6701a3fe@igalia.com> Message-ID: <7cc3cc3d-7f67-4c69-bccb-32133e1d7cba@igalia.com> On 03/05/2025 21:59, Ma?ra Canal wrote: > As more KUnit tests are introduced to evaluate the basic capabilities of > the `timedout_job()` hook, the test suite will continue to increase in > duration. To reduce the overall running time of the test suite, decrease > the scheduler's timeout for the timeout tests. > > Before this commit: > > [15:42:26] Elapsed time: 15.637s total, 0.002s configuring, 10.387s building, 5.229s running > > After this commit: > > [15:45:26] Elapsed time: 9.263s total, 0.002s configuring, 5.168s building, 4.037s running > > Signed-off-by: Ma?ra Canal > --- > drivers/gpu/drm/scheduler/tests/tests_basic.c | 10 ++++++---- > 1 file changed, 6 insertions(+), 4 deletions(-) > > diff --git a/drivers/gpu/drm/scheduler/tests/tests_basic.c b/drivers/gpu/drm/scheduler/tests/tests_basic.c > index 8f960f0fd31d0af7873f410ceba2d636f58a5474..00c691cb3c306f609684f554f17fcb54ba74cb95 100644 > --- a/drivers/gpu/drm/scheduler/tests/tests_basic.c > +++ b/drivers/gpu/drm/scheduler/tests/tests_basic.c > @@ -5,6 +5,8 @@ > > #include "sched_tests.h" > > +#define MOCK_TIMEOUT (HZ / 5) > + > /* > * DRM scheduler basic tests should check the basic functional correctness of > * the scheduler, including some very light smoke testing. More targeted tests, > @@ -28,7 +30,7 @@ static void drm_sched_basic_exit(struct kunit *test) > > static int drm_sched_timeout_init(struct kunit *test) > { > - test->priv = drm_mock_sched_new(test, HZ); > + test->priv = drm_mock_sched_new(test, MOCK_TIMEOUT); > > return 0; > } > @@ -224,17 +226,17 @@ static void drm_sched_basic_timeout(struct kunit *test) > > drm_mock_sched_job_submit(job); > > - done = drm_mock_sched_job_wait_scheduled(job, HZ); > + done = drm_mock_sched_job_wait_scheduled(job, MOCK_TIMEOUT); This wait is accounting for the fact sched->wq needs to run and call ->run_job() before job will become scheduled. It is not related to timeout handling. I was going for a safe value and I think decreasing it will not speed up the test but may cause sporadic failures. > KUNIT_ASSERT_TRUE(test, done); > > - done = drm_mock_sched_job_wait_finished(job, HZ / 2); > + done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT / 2); > KUNIT_ASSERT_FALSE(test, done); > > KUNIT_ASSERT_EQ(test, > job->flags & DRM_MOCK_SCHED_JOB_TIMEDOUT, > 0); > > - done = drm_mock_sched_job_wait_finished(job, HZ); > + done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT); > KUNIT_ASSERT_FALSE(test, done); Above two are related to timeout handling and should be safe to change. With HZ / 5 first assert could have a false negative if timeout work would run, but later than 100ms (HZ / 5 / 2). And the second a false negative if it fails to run in 300ms (HZ / 5 / 2 + HZ / 5). Neither failure sounds likely in the kunit environment so, again, I think those two are okay to speed up. Regards, Tvrtko > > KUNIT_ASSERT_EQ(test, > From mcanal at igalia.com Tue May 6 12:46:06 2025 From: mcanal at igalia.com (=?UTF-8?Q?Ma=C3=ADra_Canal?=) Date: Tue, 6 May 2025 09:46:06 -0300 Subject: [PATCH 2/8] drm/sched: Always free the job after the timeout In-Reply-To: <3fe178ec-9c16-4abc-b302-64f0077d8af4@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-2-ed0d6701a3fe@igalia.com> <3fe178ec-9c16-4abc-b302-64f0077d8af4@igalia.com> Message-ID: Hi Tvrtko, Thanks for your review! On 06/05/25 08:49, Tvrtko Ursulin wrote: > > On 03/05/2025 21:59, Ma?ra Canal wrote: >> Currently, if we add the assertions presented in this commit to the mock >> scheduler, we will see the following output: >> >> [15:47:08] ============== [PASSED] drm_sched_basic_tests ============== >> [15:47:08] ======== drm_sched_basic_timeout_tests (1 subtest) ========= >> [15:47:08] # drm_sched_basic_timeout: ASSERTION FAILED at drivers/gpu/ >> drm/scheduler/tests/tests_basic.c:246 >> [15:47:08] Expected list_empty(&sched->job_list) to be true, but is false >> [15:47:08] [FAILED] drm_sched_basic_timeout >> [15:47:08] # module: drm_sched_tests >> >> This occurs because `mock_sched_timedout_job()` doesn't properly handle >> the hang. From the DRM sched documentation, `drm_sched_stop()` and >> `drm_sched_start()` are typically used for reset recovery. If these >> functions are not used, the offending job won't be freed and should be >> freed by the caller. >> >> Currently, the mock scheduler doesn't use the functions provided by the >> API, nor does it handle the freeing of the job. As a result, the job >> isn't >> removed from the job list. > > For the record the job does gets freed via the kunit managed allocation. Sorry, I didn't express myself correctly. Indeed, it is. I meant that the DRM scheduler didn't free the job. > > It was a design choice for this test to be a *strict* unit test which > tests only a _single_ thing. And that is that the timedout_job() hook > gets called. As such the hook was implemented to satisfy that single > requirement only. > What do you think about checking that `sched->job_list` won't be empty? I wanted to add such assertion to make sure that the behavior of the timeout won't change in future (e.g. a patch makes a change that calls `free_job()` for the guilty job at timeout). Does it make sense to you? > But I also do not oppose making it test multiple things in one test per se. > >> This commit mocks a GPU reset by stopping the scheduler affected by the >> reset, waiting a couple of microseconds to mimic a hardware reset, and >> then restart the affected scheduler. >> >> Signed-off-by: Ma?ra Canal >> --- >> ? drivers/gpu/drm/scheduler/tests/mock_scheduler.c | 10 ++++++++++ >> ? drivers/gpu/drm/scheduler/tests/tests_basic.c??? |? 3 +++ >> ? 2 files changed, 13 insertions(+) >> [...] >> diff --git a/drivers/gpu/drm/scheduler/tests/tests_basic.c b/drivers/ >> gpu/drm/scheduler/tests/tests_basic.c >> index >> 7230057e0594c6246f02608f07fcb1f8d738ac75..8f960f0fd31d0af7873f410ceba2d636f58a5474 100644 >> --- a/drivers/gpu/drm/scheduler/tests/tests_basic.c >> +++ b/drivers/gpu/drm/scheduler/tests/tests_basic.c >> @@ -241,6 +241,9 @@ static void drm_sched_basic_timeout(struct kunit >> *test) >> ????????????? job->flags & DRM_MOCK_SCHED_JOB_TIMEDOUT, >> ????????????? DRM_MOCK_SCHED_JOB_TIMEDOUT); >> +??? KUNIT_ASSERT_TRUE(test, list_empty(&sched->job_list)); > > Hmm I think this assert could be racy because it appears to rely on the > free worker to run and cleanup the "finished" job in the window between > drm_mock_sched_job_wait_finished() (or drm_sched_start(), depends how > you look at it) and here. Am I missing something? From what I understand, the job is freed by the timeout worker [1] after `drm_sched_stop()` marked the job as guilty. Therefore, if the timeout was called (and we asserted that through `job->flags`), we can be sure that the job was freed. [1] https://gitlab.freedesktop.org/drm/misc/kernel/-/blob/drm-misc-next/drivers/gpu/drm/scheduler/sched_main.c#L568 Best Regards, - Ma?ra > > Regards, > > Tvrtko > >> +??? KUNIT_ASSERT_TRUE(test, list_empty(&sched->done_list)); > > +>?????? drm_mock_sched_entity_free(entity); >> ? } >> > From mcanal at igalia.com Tue May 6 12:56:27 2025 From: mcanal at igalia.com (=?UTF-8?Q?Ma=C3=ADra_Canal?=) Date: Tue, 6 May 2025 09:56:27 -0300 Subject: [PATCH 3/8] drm/sched: Reduce scheduler's timeout for timeout tests In-Reply-To: <7cc3cc3d-7f67-4c69-bccb-32133e1d7cba@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-3-ed0d6701a3fe@igalia.com> <7cc3cc3d-7f67-4c69-bccb-32133e1d7cba@igalia.com> Message-ID: Hi Tvrtko, Thanks for your review! On 06/05/25 09:03, Tvrtko Ursulin wrote: > > On 03/05/2025 21:59, Ma?ra Canal wrote: >> As more KUnit tests are introduced to evaluate the basic capabilities of >> the `timedout_job()` hook, the test suite will continue to increase in >> duration. To reduce the overall running time of the test suite, decrease >> the scheduler's timeout for the timeout tests. >> >> Before this commit: >> >> [15:42:26] Elapsed time: 15.637s total, 0.002s configuring, 10.387s >> building, 5.229s running >> >> After this commit: >> >> [15:45:26] Elapsed time: 9.263s total, 0.002s configuring, 5.168s >> building, 4.037s running >> >> Signed-off-by: Ma?ra Canal >> --- >> ? drivers/gpu/drm/scheduler/tests/tests_basic.c | 10 ++++++---- >> ? 1 file changed, 6 insertions(+), 4 deletions(-) >> >> diff --git a/drivers/gpu/drm/scheduler/tests/tests_basic.c b/drivers/ >> gpu/drm/scheduler/tests/tests_basic.c >> index >> 8f960f0fd31d0af7873f410ceba2d636f58a5474..00c691cb3c306f609684f554f17fcb54ba74cb95 100644 >> --- a/drivers/gpu/drm/scheduler/tests/tests_basic.c >> +++ b/drivers/gpu/drm/scheduler/tests/tests_basic.c >> @@ -5,6 +5,8 @@ >> ? #include "sched_tests.h" >> +#define MOCK_TIMEOUT (HZ / 5) >> + >> ? /* >> ?? * DRM scheduler basic tests should check the basic functional >> correctness of >> ?? * the scheduler, including some very light smoke testing. More >> targeted tests, >> @@ -28,7 +30,7 @@ static void drm_sched_basic_exit(struct kunit *test) >> ? static int drm_sched_timeout_init(struct kunit *test) >> ? { >> -??? test->priv = drm_mock_sched_new(test, HZ); >> +??? test->priv = drm_mock_sched_new(test, MOCK_TIMEOUT); >> ????? return 0; >> ? } >> @@ -224,17 +226,17 @@ static void drm_sched_basic_timeout(struct kunit >> *test) >> ????? drm_mock_sched_job_submit(job); >> -??? done = drm_mock_sched_job_wait_scheduled(job, HZ); >> +??? done = drm_mock_sched_job_wait_scheduled(job, MOCK_TIMEOUT); > > This wait is accounting for the fact sched->wq needs to run and call - > >run_job() before job will become scheduled. It is not related to > timeout handling. I was going for a safe value and I think decreasing it > will not speed up the test but may cause sporadic failures. I'll address it in v2. > >> ????? KUNIT_ASSERT_TRUE(test, done); >> -??? done = drm_mock_sched_job_wait_finished(job, HZ / 2); >> +??? done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT / 2); >> ????? KUNIT_ASSERT_FALSE(test, done); >> ????? KUNIT_ASSERT_EQ(test, >> ????????????? job->flags & DRM_MOCK_SCHED_JOB_TIMEDOUT, >> ????????????? 0); >> -??? done = drm_mock_sched_job_wait_finished(job, HZ); >> +??? done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT); >> ????? KUNIT_ASSERT_FALSE(test, done); > > Above two are related to timeout handling and should be safe to change. > > With HZ / 5 first assert could have a false negative if timeout work > would run, but later than 100ms (HZ / 5 / 2). And the second a false > negative if it fails to run in 300ms (HZ / 5 / 2 + HZ / 5). Neither > failure sounds likely in the kunit environment so, again, I think those > two are okay to speed up. What do you think about using a slightly bigger timeout? Maybe HZ / 4 or HZ / 2. Best Regards, - Ma?ra > > Regards, > > Tvrtko > >> ????? KUNIT_ASSERT_EQ(test, >> > From tvrtko.ursulin at igalia.com Tue May 6 13:20:04 2025 From: tvrtko.ursulin at igalia.com (Tvrtko Ursulin) Date: Tue, 6 May 2025 14:20:04 +0100 Subject: [PATCH 3/8] drm/sched: Reduce scheduler's timeout for timeout tests In-Reply-To: References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-3-ed0d6701a3fe@igalia.com> <7cc3cc3d-7f67-4c69-bccb-32133e1d7cba@igalia.com> Message-ID: <146f3943-0a94-4399-9f49-be8228a86828@igalia.com> On 06/05/2025 13:56, Ma?ra Canal wrote: > Hi Tvrtko, > > Thanks for your review! > > On 06/05/25 09:03, Tvrtko Ursulin wrote: >> >> On 03/05/2025 21:59, Ma?ra Canal wrote: >>> As more KUnit tests are introduced to evaluate the basic capabilities of >>> the `timedout_job()` hook, the test suite will continue to increase in >>> duration. To reduce the overall running time of the test suite, decrease >>> the scheduler's timeout for the timeout tests. >>> >>> Before this commit: >>> >>> [15:42:26] Elapsed time: 15.637s total, 0.002s configuring, 10.387s >>> building, 5.229s running >>> >>> After this commit: >>> >>> [15:45:26] Elapsed time: 9.263s total, 0.002s configuring, 5.168s >>> building, 4.037s running >>> >>> Signed-off-by: Ma?ra Canal >>> --- >>> ? drivers/gpu/drm/scheduler/tests/tests_basic.c | 10 ++++++---- >>> ? 1 file changed, 6 insertions(+), 4 deletions(-) >>> >>> diff --git a/drivers/gpu/drm/scheduler/tests/tests_basic.c b/drivers/ >>> gpu/drm/scheduler/tests/tests_basic.c >>> index >>> 8f960f0fd31d0af7873f410ceba2d636f58a5474..00c691cb3c306f609684f554f17fcb54ba74cb95 100644 >>> --- a/drivers/gpu/drm/scheduler/tests/tests_basic.c >>> +++ b/drivers/gpu/drm/scheduler/tests/tests_basic.c >>> @@ -5,6 +5,8 @@ >>> ? #include "sched_tests.h" >>> +#define MOCK_TIMEOUT (HZ / 5) >>> + >>> ? /* >>> ?? * DRM scheduler basic tests should check the basic functional >>> correctness of >>> ?? * the scheduler, including some very light smoke testing. More >>> targeted tests, >>> @@ -28,7 +30,7 @@ static void drm_sched_basic_exit(struct kunit *test) >>> ? static int drm_sched_timeout_init(struct kunit *test) >>> ? { >>> -??? test->priv = drm_mock_sched_new(test, HZ); >>> +??? test->priv = drm_mock_sched_new(test, MOCK_TIMEOUT); >>> ????? return 0; >>> ? } >>> @@ -224,17 +226,17 @@ static void drm_sched_basic_timeout(struct >>> kunit *test) >>> ????? drm_mock_sched_job_submit(job); >>> -??? done = drm_mock_sched_job_wait_scheduled(job, HZ); >>> +??? done = drm_mock_sched_job_wait_scheduled(job, MOCK_TIMEOUT); >> >> This wait is accounting for the fact sched->wq needs to run and call - >> ?>run_job() before job will become scheduled. It is not related to >> timeout handling. I was going for a safe value and I think decreasing >> it will not speed up the test but may cause sporadic failures. > > I'll address it in v2. > >> >>> ????? KUNIT_ASSERT_TRUE(test, done); >>> -??? done = drm_mock_sched_job_wait_finished(job, HZ / 2); >>> +??? done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT / 2); >>> ????? KUNIT_ASSERT_FALSE(test, done); >>> ????? KUNIT_ASSERT_EQ(test, >>> ????????????? job->flags & DRM_MOCK_SCHED_JOB_TIMEDOUT, >>> ????????????? 0); >>> -??? done = drm_mock_sched_job_wait_finished(job, HZ); >>> +??? done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT); >>> ????? KUNIT_ASSERT_FALSE(test, done); >> >> Above two are related to timeout handling and should be safe to change. >> >> With HZ / 5 first assert could have a false negative if timeout work >> would run, but later than 100ms (HZ / 5 / 2). And the second a false >> negative if it fails to run in 300ms (HZ / 5 / 2 + HZ / 5). Neither >> failure sounds likely in the kunit environment so, again, I think >> those two are okay to speed up. > > What do you think about using a slightly bigger timeout? Maybe HZ / 4 or > HZ / 2. I thought HZ / 5 would be safe so I'd leave it and only tweak if it turns out to be a problem. Regards, Tvrtko From tvrtko.ursulin at igalia.com Tue May 6 13:28:15 2025 From: tvrtko.ursulin at igalia.com (Tvrtko Ursulin) Date: Tue, 6 May 2025 14:28:15 +0100 Subject: [PATCH 2/8] drm/sched: Always free the job after the timeout In-Reply-To: References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-2-ed0d6701a3fe@igalia.com> <3fe178ec-9c16-4abc-b302-64f0077d8af4@igalia.com> Message-ID: <7d1e7571-8fde-40e2-8ce9-a956389ea2c0@igalia.com> On 06/05/2025 13:46, Ma?ra Canal wrote: > Hi Tvrtko, > > Thanks for your review! > > On 06/05/25 08:49, Tvrtko Ursulin wrote: >> >> On 03/05/2025 21:59, Ma?ra Canal wrote: >>> Currently, if we add the assertions presented in this commit to the mock >>> scheduler, we will see the following output: >>> >>> [15:47:08] ============== [PASSED] drm_sched_basic_tests ============== >>> [15:47:08] ======== drm_sched_basic_timeout_tests (1 subtest) ========= >>> [15:47:08] # drm_sched_basic_timeout: ASSERTION FAILED at drivers/ >>> gpu/ drm/scheduler/tests/tests_basic.c:246 >>> [15:47:08] Expected list_empty(&sched->job_list) to be true, but is >>> false >>> [15:47:08] [FAILED] drm_sched_basic_timeout >>> [15:47:08] # module: drm_sched_tests >>> >>> This occurs because `mock_sched_timedout_job()` doesn't properly handle >>> the hang. From the DRM sched documentation, `drm_sched_stop()` and >>> `drm_sched_start()` are typically used for reset recovery. If these >>> functions are not used, the offending job won't be freed and should be >>> freed by the caller. >>> >>> Currently, the mock scheduler doesn't use the functions provided by the >>> API, nor does it handle the freeing of the job. As a result, the job >>> isn't >>> removed from the job list. >> >> For the record the job does gets freed via the kunit managed allocation. > > Sorry, I didn't express myself correctly. Indeed, it is. I meant that > the DRM scheduler didn't free the job. > >> >> It was a design choice for this test to be a *strict* unit test which >> tests only a _single_ thing. And that is that the timedout_job() hook >> gets called. As such the hook was implemented to satisfy that single >> requirement only. >> > > What do you think about checking that `sched->job_list` won't be empty? > > I wanted to add such assertion to make sure that the behavior of the > timeout won't change in future (e.g. a patch makes a change that calls > `free_job()` for the guilty job at timeout). Does it make sense to you? Where would that assert be? >> But I also do not oppose making it test multiple things in one test >> per se. >> >>> This commit mocks a GPU reset by stopping the scheduler affected by the >>> reset, waiting a couple of microseconds to mimic a hardware reset, and >>> then restart the affected scheduler. >>> >>> Signed-off-by: Ma?ra Canal >>> --- >>> ? drivers/gpu/drm/scheduler/tests/mock_scheduler.c | 10 ++++++++++ >>> ? drivers/gpu/drm/scheduler/tests/tests_basic.c??? |? 3 +++ >>> ? 2 files changed, 13 insertions(+) >>> > > [...] > >>> diff --git a/drivers/gpu/drm/scheduler/tests/tests_basic.c b/drivers/ >>> gpu/drm/scheduler/tests/tests_basic.c >>> index >>> 7230057e0594c6246f02608f07fcb1f8d738ac75..8f960f0fd31d0af7873f410ceba2d636f58a5474 100644 >>> --- a/drivers/gpu/drm/scheduler/tests/tests_basic.c >>> +++ b/drivers/gpu/drm/scheduler/tests/tests_basic.c >>> @@ -241,6 +241,9 @@ static void drm_sched_basic_timeout(struct kunit >>> *test) >>> ????????????? job->flags & DRM_MOCK_SCHED_JOB_TIMEDOUT, >>> ????????????? DRM_MOCK_SCHED_JOB_TIMEDOUT); >>> +??? KUNIT_ASSERT_TRUE(test, list_empty(&sched->job_list)); >> >> Hmm I think this assert could be racy because it appears to rely on >> the free worker to run and cleanup the "finished" job in the window >> between drm_mock_sched_job_wait_finished() (or drm_sched_start(), >> depends how you look at it) and here. Am I missing something? > > From what I understand, the job is freed by the timeout worker [1] after > `drm_sched_stop()` marked the job as guilty. > > Therefore, if the timeout was called (and we asserted that through > `job->flags`), we can be sure that the job was freed. > > [1] https://gitlab.freedesktop.org/drm/misc/kernel/-/blob/drm-misc-next/ > drivers/gpu/drm/scheduler/sched_main.c#L568 Hm I thought it would end up on the dma_fence_remove_callback() == true branch in drm_sched_stop(). I gave it a quick spin locally and that indeed appears to be the case. So AFAICT it does rely on the free worker to have had executed before the assert. Regards, Tvrtko > > Best Regards, > - Ma?ra > >> >> Regards, >> >> Tvrtko >> >>> +??? KUNIT_ASSERT_TRUE(test, list_empty(&sched->done_list)); >> ?> +>?????? drm_mock_sched_entity_free(entity); >>> ? } >>> >> > From mcanal at igalia.com Tue May 6 13:38:35 2025 From: mcanal at igalia.com (=?UTF-8?Q?Ma=C3=ADra_Canal?=) Date: Tue, 6 May 2025 10:38:35 -0300 Subject: [PATCH 2/8] drm/sched: Always free the job after the timeout In-Reply-To: <7d1e7571-8fde-40e2-8ce9-a956389ea2c0@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-2-ed0d6701a3fe@igalia.com> <3fe178ec-9c16-4abc-b302-64f0077d8af4@igalia.com> <7d1e7571-8fde-40e2-8ce9-a956389ea2c0@igalia.com> Message-ID: Hi Tvrtko, On 06/05/25 10:28, Tvrtko Ursulin wrote: > > On 06/05/2025 13:46, Ma?ra Canal wrote: >> Hi Tvrtko, >> >> Thanks for your review! >> >> On 06/05/25 08:49, Tvrtko Ursulin wrote: >>> >>> On 03/05/2025 21:59, Ma?ra Canal wrote: >>>> Currently, if we add the assertions presented in this commit to the >>>> mock >>>> scheduler, we will see the following output: >>>> >>>> [15:47:08] ============== [PASSED] drm_sched_basic_tests ============== >>>> [15:47:08] ======== drm_sched_basic_timeout_tests (1 subtest) ========= >>>> [15:47:08] # drm_sched_basic_timeout: ASSERTION FAILED at drivers/ >>>> gpu/ drm/scheduler/tests/tests_basic.c:246 >>>> [15:47:08] Expected list_empty(&sched->job_list) to be true, but is >>>> false >>>> [15:47:08] [FAILED] drm_sched_basic_timeout >>>> [15:47:08] # module: drm_sched_tests >>>> >>>> This occurs because `mock_sched_timedout_job()` doesn't properly handle >>>> the hang. From the DRM sched documentation, `drm_sched_stop()` and >>>> `drm_sched_start()` are typically used for reset recovery. If these >>>> functions are not used, the offending job won't be freed and should be >>>> freed by the caller. >>>> >>>> Currently, the mock scheduler doesn't use the functions provided by the >>>> API, nor does it handle the freeing of the job. As a result, the job >>>> isn't >>>> removed from the job list. >>> >>> For the record the job does gets freed via the kunit managed allocation. >> >> Sorry, I didn't express myself correctly. Indeed, it is. I meant that >> the DRM scheduler didn't free the job. >> >>> >>> It was a design choice for this test to be a *strict* unit test which >>> tests only a _single_ thing. And that is that the timedout_job() hook >>> gets called. As such the hook was implemented to satisfy that single >>> requirement only. >>> >> >> What do you think about checking that `sched->job_list` won't be empty? >> >> I wanted to add such assertion to make sure that the behavior of the >> timeout won't change in future (e.g. a patch makes a change that calls >> `free_job()` for the guilty job at timeout). Does it make sense to you? > > Where would that assert be? > I believe it would be in the same place as this patch assertions, but instead of `KUNIT_ASSERT_TRUE(test, list_empty(&sched->job_list));`, it would be `KUNIT_ASSERT_FALSE(test, list_empty(&sched->job_list));`. But I don't feel strongly about it. I can drop the patch if you believe it's a better option. Best Regards, - Ma?ra From tvrtko.ursulin at igalia.com Tue May 6 13:58:35 2025 From: tvrtko.ursulin at igalia.com (Tvrtko Ursulin) Date: Tue, 6 May 2025 14:58:35 +0100 Subject: [PATCH 4/8] drm/sched: Add new test for DRM_GPU_SCHED_STAT_RUNNING In-Reply-To: <20250503-sched-skip-reset-v1-4-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-4-ed0d6701a3fe@igalia.com> Message-ID: <38308f2b-eaf1-4e6e-b772-690585c493c6@igalia.com> On 03/05/2025 21:59, Ma?ra Canal wrote: > Add a test to submit a single job against a scheduler with the timeout > configured and verify that if the job is still running, the timeout > handler will skip the reset and allow the job to complete. > > Signed-off-by: Ma?ra Canal > --- > drivers/gpu/drm/scheduler/tests/mock_scheduler.c | 3 ++ > drivers/gpu/drm/scheduler/tests/tests_basic.c | 44 ++++++++++++++++++++++++ > 2 files changed, 47 insertions(+) > > diff --git a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c > index e9af202d84bd55ea5cc048215e39f5407bc84458..9d594cb5bf567be25e018ddbcd28b70a7e994260 100644 > --- a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c > +++ b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c > @@ -211,6 +211,9 @@ mock_sched_timedout_job(struct drm_sched_job *sched_job) > > job->flags |= DRM_MOCK_SCHED_JOB_TIMEDOUT; > > + if (job->finish_at && ktime_before(ktime_get(), job->finish_at)) > + return DRM_GPU_SCHED_STAT_RUNNING; Hmm this would prevent testing timeout handling with mock jobs with duration set for any future test. It works, but I am thinking if a more explicit way wouldn't be better. For example to add a new job flag like DRM_MOCK_SCHED_JOB_DONTRESET or similar? That way test could use the timing insensitive drm_mock_sched_advance() and have explicit control of the execution. If you don't mind I think I would prefer that. Regards, Tvrtko > + > drm_sched_stop(&sched->base, &job->base); > > usleep_range(200, 500); > diff --git a/drivers/gpu/drm/scheduler/tests/tests_basic.c b/drivers/gpu/drm/scheduler/tests/tests_basic.c > index 00c691cb3c306f609684f554f17fcb54ba74cb95..669a211b216ee298544ac237abb866077d856586 100644 > --- a/drivers/gpu/drm/scheduler/tests/tests_basic.c > +++ b/drivers/gpu/drm/scheduler/tests/tests_basic.c > @@ -249,8 +249,52 @@ static void drm_sched_basic_timeout(struct kunit *test) > drm_mock_sched_entity_free(entity); > } > > +static void drm_sched_skip_reset(struct kunit *test) > +{ > + struct drm_mock_scheduler *sched = test->priv; > + struct drm_mock_sched_entity *entity; > + struct drm_mock_sched_job *job; > + bool done; > + > + /* > + * Submit a single job against a scheduler with the timeout configured > + * and verify that if the job is still running, the timeout handler > + * will skip the reset and allow the job to complete. > + */ > + > + entity = drm_mock_sched_entity_new(test, > + DRM_SCHED_PRIORITY_NORMAL, > + sched); > + job = drm_mock_sched_job_new(test, entity); > + > + drm_mock_sched_job_set_duration_us(job, jiffies_to_usecs(2 * MOCK_TIMEOUT)); > + drm_mock_sched_job_submit(job); > + > + done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT); > + KUNIT_ASSERT_FALSE(test, done); > + > + KUNIT_ASSERT_EQ(test, > + job->flags & DRM_MOCK_SCHED_JOB_TIMEDOUT, > + DRM_MOCK_SCHED_JOB_TIMEDOUT); > + > + KUNIT_ASSERT_FALSE(test, list_empty(&sched->job_list)); > + > + done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT); > + KUNIT_ASSERT_TRUE(test, done); > + > + KUNIT_ASSERT_EQ(test, > + job->flags & DRM_MOCK_SCHED_JOB_TIMEDOUT, > + DRM_MOCK_SCHED_JOB_TIMEDOUT); > + > + KUNIT_ASSERT_TRUE(test, list_empty(&sched->job_list)); > + KUNIT_ASSERT_TRUE(test, list_empty(&sched->done_list)); > + > + drm_mock_sched_entity_free(entity); > +} > + > static struct kunit_case drm_sched_timeout_tests[] = { > KUNIT_CASE(drm_sched_basic_timeout), > + KUNIT_CASE(drm_sched_skip_reset), > {} > }; > > From tvrtko.ursulin at igalia.com Tue May 6 14:18:17 2025 From: tvrtko.ursulin at igalia.com (Tvrtko Ursulin) Date: Tue, 6 May 2025 15:18:17 +0100 Subject: [PATCH 2/8] drm/sched: Always free the job after the timeout In-Reply-To: References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-2-ed0d6701a3fe@igalia.com> <3fe178ec-9c16-4abc-b302-64f0077d8af4@igalia.com> <7d1e7571-8fde-40e2-8ce9-a956389ea2c0@igalia.com> Message-ID: <41d87d5e-62a4-4b2a-a626-a90caed50877@igalia.com> On 06/05/2025 14:38, Ma?ra Canal wrote: > Hi Tvrtko, > > On 06/05/25 10:28, Tvrtko Ursulin wrote: >> >> On 06/05/2025 13:46, Ma?ra Canal wrote: >>> Hi Tvrtko, >>> >>> Thanks for your review! >>> >>> On 06/05/25 08:49, Tvrtko Ursulin wrote: >>>> >>>> On 03/05/2025 21:59, Ma?ra Canal wrote: >>>>> Currently, if we add the assertions presented in this commit to the >>>>> mock >>>>> scheduler, we will see the following output: >>>>> >>>>> [15:47:08] ============== [PASSED] drm_sched_basic_tests >>>>> ============== >>>>> [15:47:08] ======== drm_sched_basic_timeout_tests (1 subtest) >>>>> ========= >>>>> [15:47:08] # drm_sched_basic_timeout: ASSERTION FAILED at drivers/ >>>>> gpu/ drm/scheduler/tests/tests_basic.c:246 >>>>> [15:47:08] Expected list_empty(&sched->job_list) to be true, but is >>>>> false >>>>> [15:47:08] [FAILED] drm_sched_basic_timeout >>>>> [15:47:08] # module: drm_sched_tests >>>>> >>>>> This occurs because `mock_sched_timedout_job()` doesn't properly >>>>> handle >>>>> the hang. From the DRM sched documentation, `drm_sched_stop()` and >>>>> `drm_sched_start()` are typically used for reset recovery. If these >>>>> functions are not used, the offending job won't be freed and should be >>>>> freed by the caller. >>>>> >>>>> Currently, the mock scheduler doesn't use the functions provided by >>>>> the >>>>> API, nor does it handle the freeing of the job. As a result, the >>>>> job isn't >>>>> removed from the job list. >>>> >>>> For the record the job does gets freed via the kunit managed >>>> allocation. >>> >>> Sorry, I didn't express myself correctly. Indeed, it is. I meant that >>> the DRM scheduler didn't free the job. >>> >>>> >>>> It was a design choice for this test to be a *strict* unit test >>>> which tests only a _single_ thing. And that is that the >>>> timedout_job() hook gets called. As such the hook was implemented to >>>> satisfy that single requirement only. >>>> >>> >>> What do you think about checking that `sched->job_list` won't be empty? >>> >>> I wanted to add such assertion to make sure that the behavior of the >>> timeout won't change in future (e.g. a patch makes a change that calls >>> `free_job()` for the guilty job at timeout). Does it make sense to you? >> >> Where would that assert be? >> > > I believe it would be in the same place as this patch assertions, but > instead of `KUNIT_ASSERT_TRUE(test, list_empty(&sched->job_list));`, it > would be `KUNIT_ASSERT_FALSE(test, list_empty(&sched->job_list));`. > > But I don't feel strongly about it. I can drop the patch if you believe > it's a better option. I don't mind this patch, I was just explaining how the current test was deliberately testing a single thing. But we can change it to test more, that's fine. In this case it would go from testing that the timeout callback fires to testing the callback fires and something else happens. Key is to define the "something else" part so it is not timing sensitive. The drm_sched_stop+delay+drm_sched_start approach would perhaps work if you would signal the job with an errno set before drm_sched_stop? Then it would hit the "free_guilty" path in drm_sched_stop and wouldn't be timing sensitive. As long as there aren't any locking considerations I am missing. That could then safely have the two list_empty asserts. Regards, Tvrtko From matthew.brost at intel.com Tue May 6 14:32:13 2025 From: matthew.brost at intel.com (Matthew Brost) Date: Tue, 6 May 2025 07:32:13 -0700 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> Message-ID: On Mon, May 05, 2025 at 07:41:09PM -0700, Matthew Brost wrote: > On Sat, May 03, 2025 at 05:59:52PM -0300, Ma?ra Canal wrote: > > When the DRM scheduler times out, it's possible that the GPU isn't hung; > > instead, a job may still be running, and there may be no valid reason to > > reset the hardware. This can occur in two situations: > > > > 1. The GPU exposes some mechanism that ensures the GPU is still making > > progress. By checking this mechanism, we can safely skip the reset, > > rearm the timeout, and allow the job to continue running until > > completion. This is the case for v3d and Etnaviv. > > 2. TDR has fired before the IRQ that signals the fence. Consequently, > > the job actually finishes, but it triggers a timeout before signaling > > the completion fence. > > > > We have both of these cases in Xe too. We implement the requeuing in Xe > via driver side function - xe_sched_add_pending_job but this looks > better and will make use of this. > > > These two scenarios are problematic because we remove the job from the > > `sched->pending_list` before calling `sched->ops->timedout_job()`. This > > means that when the job finally signals completion (e.g. in the IRQ > > handler), the scheduler won't call `sched->ops->free_job()`. As a result, > > the job and its resources won't be freed, leading to a memory leak. > > > > To resolve this issue, we create a new `drm_gpu_sched_stat` that allows a > > driver to skip the reset. This new status will indicate that the job > > should be reinserted into the pending list, and the driver will still > > signal its completion. > > > > Signed-off-by: Ma?ra Canal > > Reviewed-by: Matthew Brost > Wait - nevermind I think one issue is below. > > --- > > drivers/gpu/drm/scheduler/sched_main.c | 14 ++++++++++++++ > > include/drm/gpu_scheduler.h | 2 ++ > > 2 files changed, 16 insertions(+) > > > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c > > index 829579c41c6b5d8b2abce5ad373c7017469b7680..68ca827d77e32187a034309f881135dbc639a9b4 100644 > > --- a/drivers/gpu/drm/scheduler/sched_main.c > > +++ b/drivers/gpu/drm/scheduler/sched_main.c > > @@ -568,6 +568,17 @@ static void drm_sched_job_timedout(struct work_struct *work) > > job->sched->ops->free_job(job); > > sched->free_guilty = false; > > } > > + > > + /* > > + * If the driver indicated that the GPU is still running and wants to skip > > + * the reset, reinsert the job back into the pending list and realarm the > > + * timeout. > > + */ > > + if (status == DRM_GPU_SCHED_STAT_RUNNING) { > > + spin_lock(&sched->job_list_lock); > > + list_add(&job->list, &sched->pending_list); > > + spin_unlock(&sched->job_list_lock); > > + } I think you need to requeue free_job wq here. It is possible the free_job wq ran, didn't find a job, goes to sleep, then we add a signaled job here which will never get freed. Matt > > } else { > > spin_unlock(&sched->job_list_lock); > > } > > @@ -590,6 +601,9 @@ static void drm_sched_job_timedout(struct work_struct *work) > > * This function is typically used for reset recovery (see the docu of > > * drm_sched_backend_ops.timedout_job() for details). Do not call it for > > * scheduler teardown, i.e., before calling drm_sched_fini(). > > + * > > + * As it's used for reset recovery, drm_sched_stop() shouldn't be called > > + * if the scheduler skipped the timeout (DRM_SCHED_STAT_RUNNING). > > */ > > void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad) > > { > > diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h > > index 1a7e377d4cbb4fc12ed93c548b236970217945e8..fe9043b6d43141bee831b5fc16b927202a507d51 100644 > > --- a/include/drm/gpu_scheduler.h > > +++ b/include/drm/gpu_scheduler.h > > @@ -389,11 +389,13 @@ struct drm_sched_job { > > * @DRM_GPU_SCHED_STAT_NONE: Reserved. Do not use. > > * @DRM_GPU_SCHED_STAT_NOMINAL: Operation succeeded. > > * @DRM_GPU_SCHED_STAT_ENODEV: Error: Device is not available anymore. > > + * @DRM_GPU_SCHED_STAT_RUNNING: GPU is still running, so skip the reset. > > */ > > enum drm_gpu_sched_stat { > > DRM_GPU_SCHED_STAT_NONE, > > DRM_GPU_SCHED_STAT_NOMINAL, > > DRM_GPU_SCHED_STAT_ENODEV, > > + DRM_GPU_SCHED_STAT_RUNNING, > > }; > > > > /** > > > > -- > > 2.49.0 > > From tomeu at tomeuvizoso.net Wed May 7 11:21:30 2025 From: tomeu at tomeuvizoso.net (Tomeu Vizoso) Date: Wed, 7 May 2025 13:21:30 +0200 Subject: [PATCH] drm/etnaviv: Fix flush sequence logic Message-ID: <20250507112131.3686966-1-tomeu@tomeuvizoso.net> We should be comparing the last submitted sequence number with that of the address space we may be switching to. And we should be using the latter as the last submitted sequence number afterwards. Signed-off-by: Tomeu Vizoso --- drivers/gpu/drm/etnaviv/etnaviv_buffer.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c index b13a17276d07..865b07b14b38 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c @@ -347,7 +347,7 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, u32 link_target, link_dwords; bool switch_context = gpu->exec_state != exec_state; bool switch_mmu_context = gpu->mmu_context != mmu_context; - unsigned int new_flush_seq = READ_ONCE(gpu->mmu_context->flush_seq); + unsigned int new_flush_seq = READ_ONCE(mmu_context->flush_seq); bool need_flush = switch_mmu_context || gpu->flush_seq != new_flush_seq; bool has_blt = !!(gpu->identity.minor_features5 & chipMinorFeatures5_BLT_ENGINE); @@ -399,6 +399,7 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, struct etnaviv_iommu_context *old_context = gpu->mmu_context; gpu->mmu_context = etnaviv_iommu_context_get(mmu_context); + gpu->flush_seq = new_flush_seq; etnaviv_iommu_context_put(old_context); } -- 2.49.0 From festevam at gmail.com Wed May 7 11:34:23 2025 From: festevam at gmail.com (Fabio Estevam) Date: Wed, 7 May 2025 08:34:23 -0300 Subject: [PATCH] drm/etnaviv: Fix flush sequence logic In-Reply-To: <20250507112131.3686966-1-tomeu@tomeuvizoso.net> References: <20250507112131.3686966-1-tomeu@tomeuvizoso.net> Message-ID: Hi Tomeu, On Wed, May 7, 2025 at 8:22?AM Tomeu Vizoso wrote: > > We should be comparing the last submitted sequence number with that of > the address space we may be switching to. > > And we should be using the latter as the last submitted sequence number > afterwards. > > Signed-off-by: Tomeu Vizoso Does this need a Fixes tag? From l.stach at pengutronix.de Wed May 7 12:07:22 2025 From: l.stach at pengutronix.de (Lucas Stach) Date: Wed, 07 May 2025 14:07:22 +0200 Subject: [PATCH] drm/etnaviv: Fix flush sequence logic In-Reply-To: <20250507112131.3686966-1-tomeu@tomeuvizoso.net> References: <20250507112131.3686966-1-tomeu@tomeuvizoso.net> Message-ID: <05552657cac9395b83d454e2ede05271d4c15abc.camel@pengutronix.de> Am Mittwoch, dem 07.05.2025 um 13:21 +0200 schrieb Tomeu Vizoso: > We should be comparing the last submitted sequence number with that of > the address space we may be switching to. > > And we should be using the latter as the last submitted sequence number > afterwards. > > Signed-off-by: Tomeu Vizoso > --- > drivers/gpu/drm/etnaviv/etnaviv_buffer.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c > index b13a17276d07..865b07b14b38 100644 > --- a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c > +++ b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c > @@ -347,7 +347,7 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, > u32 link_target, link_dwords; > bool switch_context = gpu->exec_state != exec_state; > bool switch_mmu_context = gpu->mmu_context != mmu_context; > - unsigned int new_flush_seq = READ_ONCE(gpu->mmu_context->flush_seq); > + unsigned int new_flush_seq = READ_ONCE(mmu_context->flush_seq); > bool need_flush = switch_mmu_context || gpu->flush_seq != new_flush_seq; > bool has_blt = !!(gpu->identity.minor_features5 & > chipMinorFeatures5_BLT_ENGINE); > @@ -399,6 +399,7 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, > struct etnaviv_iommu_context *old_context = gpu->mmu_context; > > gpu->mmu_context = etnaviv_iommu_context_get(mmu_context); > + gpu->flush_seq = new_flush_seq; As I said in IRC: this should not be necessary. If we switch to a new MMU context, need_flush will also be set. The code under the need_flush condition already includes this exact line. Regards, Lucas From mcanal at igalia.com Wed May 7 12:33:31 2025 From: mcanal at igalia.com (=?UTF-8?Q?Ma=C3=ADra_Canal?=) Date: Wed, 7 May 2025 09:33:31 -0300 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> Message-ID: <4020cf8b-3524-46c9-a082-adaf4c1797c2@igalia.com> Hi Tvrtko, Thanks for the review! On 06/05/25 08:32, Tvrtko Ursulin wrote: > > On 03/05/2025 21:59, Ma?ra Canal wrote: >> When the DRM scheduler times out, it's possible that the GPU isn't hung; >> instead, a job may still be running, and there may be no valid reason to >> reset the hardware. This can occur in two situations: >> >> ?? 1. The GPU exposes some mechanism that ensures the GPU is still making >> ????? progress. By checking this mechanism, we can safely skip the reset, >> ????? rearm the timeout, and allow the job to continue running until >> ????? completion. This is the case for v3d and Etnaviv. >> ?? 2. TDR has fired before the IRQ that signals the fence. Consequently, >> ????? the job actually finishes, but it triggers a timeout before >> signaling >> ????? the completion fence. >> >> These two scenarios are problematic because we remove the job from the >> `sched->pending_list` before calling `sched->ops->timedout_job()`. This >> means that when the job finally signals completion (e.g. in the IRQ >> handler), the scheduler won't call `sched->ops->free_job()`. As a result, >> the job and its resources won't be freed, leading to a memory leak. >> >> To resolve this issue, we create a new `drm_gpu_sched_stat` that allows a >> driver to skip the reset. This new status will indicate that the job >> should be reinserted into the pending list, and the driver will still >> signal its completion. > [...] >> diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h >> index >> 1a7e377d4cbb4fc12ed93c548b236970217945e8..fe9043b6d43141bee831b5fc16b927202a507d51 100644 >> --- a/include/drm/gpu_scheduler.h >> +++ b/include/drm/gpu_scheduler.h >> @@ -389,11 +389,13 @@ struct drm_sched_job { >> ?? * @DRM_GPU_SCHED_STAT_NONE: Reserved. Do not use. >> ?? * @DRM_GPU_SCHED_STAT_NOMINAL: Operation succeeded. >> ?? * @DRM_GPU_SCHED_STAT_ENODEV: Error: Device is not available anymore. >> + * @DRM_GPU_SCHED_STAT_RUNNING: GPU is still running, so skip the reset. > > s/GPU/job/ ? > >> ?? */ >> ? enum drm_gpu_sched_stat { >> ????? DRM_GPU_SCHED_STAT_NONE, >> ????? DRM_GPU_SCHED_STAT_NOMINAL, >> ????? DRM_GPU_SCHED_STAT_ENODEV, >> +??? DRM_GPU_SCHED_STAT_RUNNING, > > I am wondering if we could make it more obvious what is the difference > between "nominal" and "running" and from whose point of view should > those statuses be considered. > > So far we have "nominal" which means scheduler/hardware is working fine > but the job may or may have not been cancelled. With "running" we kind > of split it into two sub-statuses and it would be nice for that to be > intuitively visible from the naming. But I struggle to suggest an > elegant name while preserving nominal as is. I was thinking: how about changing DRM_GPU_SCHED_STAT_NOMINAL to DRM_GPU_SCHED_STAT_RESET (the hardware is fine, but we reset it)? Then, when we skip the reset, we would have DRM_GPU_SCHED_STAT_NOMINAL (which means the hardware is fine and we didn't reset it). I'm open to other suggestions. > > Thinking out loud here - perhaps that is pointing towards an alternative > that instead of a new status, a new helper to re-insert the single job > (like drm_sched_resubmit_job(sched, job)) would fit better? Although it > would be more churn. > Although your solution might be more elegant, I'm worried that such a function could be used improperly by new users (e.g. being called in contexts other than `timedout_job()`). I'd prefer to have a new status as it'll be use solely for `timedout_job()` (making it harder for users to use it inappropriately). With the addition of Matthew's feedback (calling `drm_sched_run_free_queue()` after adding the job to the pending list), I think it makes even more sense to keep it inside the timeout function. I hope others can chime in and give their opinions about your idea. Best Regards, - Ma?ra > Regards, > > Tvrtko > >> ? }; >> ? /** >> > From tvrtko.ursulin at igalia.com Wed May 7 12:50:33 2025 From: tvrtko.ursulin at igalia.com (Tvrtko Ursulin) Date: Wed, 7 May 2025 13:50:33 +0100 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: <4020cf8b-3524-46c9-a082-adaf4c1797c2@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> <4020cf8b-3524-46c9-a082-adaf4c1797c2@igalia.com> Message-ID: <95daf571-fa56-4e49-9c45-adb3932aecdb@igalia.com> On 07/05/2025 13:33, Ma?ra Canal wrote: > Hi Tvrtko, > > Thanks for the review! > > On 06/05/25 08:32, Tvrtko Ursulin wrote: >> >> On 03/05/2025 21:59, Ma?ra Canal wrote: >>> When the DRM scheduler times out, it's possible that the GPU isn't hung; >>> instead, a job may still be running, and there may be no valid reason to >>> reset the hardware. This can occur in two situations: >>> >>> ?? 1. The GPU exposes some mechanism that ensures the GPU is still >>> making >>> ????? progress. By checking this mechanism, we can safely skip the >>> reset, >>> ????? rearm the timeout, and allow the job to continue running until >>> ????? completion. This is the case for v3d and Etnaviv. >>> ?? 2. TDR has fired before the IRQ that signals the fence. Consequently, >>> ????? the job actually finishes, but it triggers a timeout before >>> signaling >>> ????? the completion fence. >>> >>> These two scenarios are problematic because we remove the job from the >>> `sched->pending_list` before calling `sched->ops->timedout_job()`. This >>> means that when the job finally signals completion (e.g. in the IRQ >>> handler), the scheduler won't call `sched->ops->free_job()`. As a >>> result, >>> the job and its resources won't be freed, leading to a memory leak. >>> >>> To resolve this issue, we create a new `drm_gpu_sched_stat` that >>> allows a >>> driver to skip the reset. This new status will indicate that the job >>> should be reinserted into the pending list, and the driver will still >>> signal its completion. >> > > [...] > >>> diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h >>> index >>> 1a7e377d4cbb4fc12ed93c548b236970217945e8..fe9043b6d43141bee831b5fc16b927202a507d51 100644 >>> --- a/include/drm/gpu_scheduler.h >>> +++ b/include/drm/gpu_scheduler.h >>> @@ -389,11 +389,13 @@ struct drm_sched_job { >>> ?? * @DRM_GPU_SCHED_STAT_NONE: Reserved. Do not use. >>> ?? * @DRM_GPU_SCHED_STAT_NOMINAL: Operation succeeded. >>> ?? * @DRM_GPU_SCHED_STAT_ENODEV: Error: Device is not available anymore. >>> + * @DRM_GPU_SCHED_STAT_RUNNING: GPU is still running, so skip the >>> reset. >> >> s/GPU/job/ ? >> >>> ?? */ >>> ? enum drm_gpu_sched_stat { >>> ????? DRM_GPU_SCHED_STAT_NONE, >>> ????? DRM_GPU_SCHED_STAT_NOMINAL, >>> ????? DRM_GPU_SCHED_STAT_ENODEV, >>> +??? DRM_GPU_SCHED_STAT_RUNNING, >> >> I am wondering if we could make it more obvious what is the difference >> between "nominal" and "running" and from whose point of view should >> those statuses be considered. > > > So far we have "nominal" which means scheduler/hardware is working > fine >> but the job may or may have not been cancelled. With "running" we kind >> of split it into two sub-statuses and it would be nice for that to be >> intuitively visible from the naming. But I struggle to suggest an >> elegant name while preserving nominal as is. > > I was thinking: how about changing DRM_GPU_SCHED_STAT_NOMINAL to > DRM_GPU_SCHED_STAT_RESET (the hardware is fine, but we reset it)? > > Then, when we skip the reset, we would have DRM_GPU_SCHED_STAT_NOMINAL > (which means the hardware is fine and we didn't reset it). > > I'm open to other suggestions. DRM_GPU_SCHED_STAT_RESET sounds like a good name and seems to paint a consistent story between running - reset - enodev. >> Thinking out loud here - perhaps that is pointing towards an >> alternative that instead of a new status, a new helper to re-insert >> the single job (like drm_sched_resubmit_job(sched, job)) would fit >> better? Although it would be more churn. >> > > Although your solution might be more elegant, I'm worried that such a > function could be used improperly by new users (e.g. being called in > contexts other than `timedout_job()`). We could call it drm_sched_untimedout_job(). > I'd prefer to have a new status as it'll be use solely for > `timedout_job()` (making it harder for users to use it inappropriately). > With the addition of Matthew's feedback (calling > `drm_sched_run_free_queue()` after adding the job to the pending list), > I think it makes even more sense to keep it inside the timeout function. > > I hope others can chime in and give their opinions about your idea. Yeah - Philipp - Danilo - what do you prefer? Third enum with or a new helper? Regards, Tvrtko From christian.gmeiner at gmail.com Wed May 7 13:07:34 2025 From: christian.gmeiner at gmail.com (Christian Gmeiner) Date: Wed, 7 May 2025 15:07:34 +0200 Subject: [PATCH] drm/etnaviv: Fix flush sequence logic In-Reply-To: References: <20250507112131.3686966-1-tomeu@tomeuvizoso.net> Message-ID: > Hi Tomeu, > > On Wed, May 7, 2025 at 8:22?AM Tomeu Vizoso wrote: > > > > We should be comparing the last submitted sequence number with that of > > the address space we may be switching to. > > > > And we should be using the latter as the last submitted sequence number > > afterwards. > > > > Signed-off-by: Tomeu Vizoso > > Does this need a Fixes tag? Yes - based on the READ_ONCE(..) line, it should be Fixes: 27b67278e007 ("drm/etnaviv: rework MMU handling") -- greets -- Christian Gmeiner, MSc https://christian-gmeiner.info/privacypolicy From tomeu at tomeuvizoso.net Thu May 8 14:56:24 2025 From: tomeu at tomeuvizoso.net (Tomeu Vizoso) Date: Thu, 8 May 2025 16:56:24 +0200 Subject: [PATCH v2] drm/etnaviv: Fix flush sequence logic In-Reply-To: <20250507112131.3686966-1-tomeu@tomeuvizoso.net> References: <20250507112131.3686966-1-tomeu@tomeuvizoso.net> Message-ID: <20250508145624.4154317-1-tomeu@tomeuvizoso.net> We should be comparing the last submitted sequence number with that of the address space we may be switching to. Fixes: 27b67278e007 ("drm/etnaviv: rework MMU handling") Signed-off-by: Tomeu Vizoso --- drivers/gpu/drm/etnaviv/etnaviv_buffer.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c index b13a17276d07..88385dc3b30d 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c @@ -347,7 +347,7 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, u32 link_target, link_dwords; bool switch_context = gpu->exec_state != exec_state; bool switch_mmu_context = gpu->mmu_context != mmu_context; - unsigned int new_flush_seq = READ_ONCE(gpu->mmu_context->flush_seq); + unsigned int new_flush_seq = READ_ONCE(mmu_context->flush_seq); bool need_flush = switch_mmu_context || gpu->flush_seq != new_flush_seq; bool has_blt = !!(gpu->identity.minor_features5 & chipMinorFeatures5_BLT_ENGINE); -- 2.49.0 From steven.price at arm.com Thu May 8 15:04:25 2025 From: steven.price at arm.com (Steven Price) Date: Thu, 8 May 2025 16:04:25 +0100 Subject: [PATCH 8/8] drm/panfrost: Use DRM_GPU_SCHED_STAT_RUNNING to skip the reset In-Reply-To: <20250503-sched-skip-reset-v1-8-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-8-ed0d6701a3fe@igalia.com> Message-ID: <6e2fec36-8631-4ba6-8b66-d8b7c8cba5f6@arm.com> On 03/05/2025 21:59, Ma?ra Canal wrote: > Panfrost can skip the reset if TDR has fired before the IRQ handler. > Currently, since Panfrost doesn't take any action on these scenarios, the > job is being leaked, considering that `free_job()` won't be called. > > To avoid such leaks, use the DRM_GPU_SCHED_STAT_RUNNING status to skip the > reset and rearm the timer. > > Signed-off-by: Ma?ra Canal Reviewed-by: Steven Price > --- > drivers/gpu/drm/panfrost/panfrost_job.c | 8 ++++---- > 1 file changed, 4 insertions(+), 4 deletions(-) > > diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c > index 5657106c2f7d0a0ca6162850767f58f3200cce13..2948d5c02115544a0e0babffd850f1506152849d 100644 > --- a/drivers/gpu/drm/panfrost/panfrost_job.c > +++ b/drivers/gpu/drm/panfrost/panfrost_job.c > @@ -751,11 +751,11 @@ static enum drm_gpu_sched_stat panfrost_job_timedout(struct drm_sched_job > int js = panfrost_job_get_slot(job); > > /* > - * If the GPU managed to complete this jobs fence, the timeout is > - * spurious. Bail out. > + * If the GPU managed to complete this jobs fence, TDR has fired before > + * IRQ and the timeout is spurious. Bail out. > */ > if (dma_fence_is_signaled(job->done_fence)) > - return DRM_GPU_SCHED_STAT_NOMINAL; > + return DRM_GPU_SCHED_STAT_RUNNING; > > /* > * Panfrost IRQ handler may take a long time to process an interrupt > @@ -770,7 +770,7 @@ static enum drm_gpu_sched_stat panfrost_job_timedout(struct drm_sched_job > > if (dma_fence_is_signaled(job->done_fence)) { > dev_warn(pfdev->dev, "unexpectedly high interrupt latency\n"); > - return DRM_GPU_SCHED_STAT_NOMINAL; > + return DRM_GPU_SCHED_STAT_RUNNING; > } > > dev_err(pfdev->dev, "gpu sched timeout, js=%d, config=0x%x, status=0x%x, head=0x%x, tail=0x%x, sched_job=%p", > From l.stach at pengutronix.de Thu May 8 17:08:04 2025 From: l.stach at pengutronix.de (Lucas Stach) Date: Thu, 08 May 2025 19:08:04 +0200 Subject: [PATCH v2] drm/etnaviv: Fix flush sequence logic In-Reply-To: <20250508145624.4154317-1-tomeu@tomeuvizoso.net> References: <20250507112131.3686966-1-tomeu@tomeuvizoso.net> <20250508145624.4154317-1-tomeu@tomeuvizoso.net> Message-ID: Am Donnerstag, dem 08.05.2025 um 16:56 +0200 schrieb Tomeu Vizoso: > We should be comparing the last submitted sequence number with that of > the address space we may be switching to. > This isn't the relevant change here though: if we switch the address space, the comparison is moot, as we do a full flush on AS switch anyway. The relevant change is that with the old code we would record the flush sequence of the AS we switch away from as the current flush sequence, so we might miss a necessary flush on the next submission if that one doesn't require a AS switch, but would only flush based on sequence mismatch. Mind if I rewrite the commit message along those lines while applying? Regards, Lucas > Fixes: 27b67278e007 ("drm/etnaviv: rework MMU handling") > Signed-off-by: Tomeu Vizoso > --- > drivers/gpu/drm/etnaviv/etnaviv_buffer.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c > index b13a17276d07..88385dc3b30d 100644 > --- a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c > +++ b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c > @@ -347,7 +347,7 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, > u32 link_target, link_dwords; > bool switch_context = gpu->exec_state != exec_state; > bool switch_mmu_context = gpu->mmu_context != mmu_context; > - unsigned int new_flush_seq = READ_ONCE(gpu->mmu_context->flush_seq); > + unsigned int new_flush_seq = READ_ONCE(mmu_context->flush_seq); > bool need_flush = switch_mmu_context || gpu->flush_seq != new_flush_seq; > bool has_blt = !!(gpu->identity.minor_features5 & > chipMinorFeatures5_BLT_ENGINE); From tomeu at tomeuvizoso.net Thu May 8 19:11:48 2025 From: tomeu at tomeuvizoso.net (Tomeu Vizoso) Date: Thu, 8 May 2025 21:11:48 +0200 Subject: [PATCH v2] drm/etnaviv: Fix flush sequence logic In-Reply-To: References: <20250507112131.3686966-1-tomeu@tomeuvizoso.net> <20250508145624.4154317-1-tomeu@tomeuvizoso.net> Message-ID: On Thu, May 8, 2025 at 7:08?PM Lucas Stach wrote: > > Am Donnerstag, dem 08.05.2025 um 16:56 +0200 schrieb Tomeu Vizoso: > > We should be comparing the last submitted sequence number with that of > > the address space we may be switching to. > > > This isn't the relevant change here though: if we switch the address > space, the comparison is moot, as we do a full flush on AS switch > anyway. The relevant change is that with the old code we would record > the flush sequence of the AS we switch away from as the current flush > sequence, so we might miss a necessary flush on the next submission if > that one doesn't require a AS switch, but would only flush based on > sequence mismatch. Ah, you are right. > Mind if I rewrite the commit message along those lines while applying? Sure, no problem. Thanks, Tomeu > Regards, > Lucas > > > Fixes: 27b67278e007 ("drm/etnaviv: rework MMU handling") > > Signed-off-by: Tomeu Vizoso > > --- > > drivers/gpu/drm/etnaviv/etnaviv_buffer.c | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c > > index b13a17276d07..88385dc3b30d 100644 > > --- a/drivers/gpu/drm/etnaviv/etnaviv_buffer.c > > +++ b/drivers/gpu/drm/etnaviv/etnaviv_buffer.c > > @@ -347,7 +347,7 @@ void etnaviv_buffer_queue(struct etnaviv_gpu *gpu, u32 exec_state, > > u32 link_target, link_dwords; > > bool switch_context = gpu->exec_state != exec_state; > > bool switch_mmu_context = gpu->mmu_context != mmu_context; > > - unsigned int new_flush_seq = READ_ONCE(gpu->mmu_context->flush_seq); > > + unsigned int new_flush_seq = READ_ONCE(mmu_context->flush_seq); > > bool need_flush = switch_mmu_context || gpu->flush_seq != new_flush_seq; > > bool has_blt = !!(gpu->identity.minor_features5 & > > chipMinorFeatures5_BLT_ENGINE); > From phasta at mailbox.org Mon May 12 09:24:48 2025 From: phasta at mailbox.org (Philipp Stanner) Date: Mon, 12 May 2025 11:24:48 +0200 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: <95daf571-fa56-4e49-9c45-adb3932aecdb@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> <4020cf8b-3524-46c9-a082-adaf4c1797c2@igalia.com> <95daf571-fa56-4e49-9c45-adb3932aecdb@igalia.com> Message-ID: <73e6d7c4d216456f10960b5409de46c669bdc3ce.camel@mailbox.org> On Wed, 2025-05-07 at 13:50 +0100, Tvrtko Ursulin wrote: > > On 07/05/2025 13:33, Ma?ra Canal wrote: > > Hi Tvrtko, > > > > Thanks for the review! > > > > On 06/05/25 08:32, Tvrtko Ursulin wrote: > > > > > > On 03/05/2025 21:59, Ma?ra Canal wrote: > > > > When the DRM scheduler times out, it's possible that the GPU > > > > isn't hung; > > > > instead, a job may still be running, and there may be no valid > > > > reason to > > > > reset the hardware. This can occur in two situations: > > > > > > > > ?? 1. The GPU exposes some mechanism that ensures the GPU is > > > > still > > > > making > > > > ????? progress. By checking this mechanism, we can safely skip > > > > the > > > > reset, > > > > ????? rearm the timeout, and allow the job to continue running > > > > until > > > > ????? completion. This is the case for v3d and Etnaviv. > > > > ?? 2. TDR has fired before the IRQ that signals the fence. > > > > Consequently, > > > > ????? the job actually finishes, but it triggers a timeout > > > > before > > > > signaling > > > > ????? the completion fence. > > > > > > > > These two scenarios are problematic because we remove the job > > > > from the > > > > `sched->pending_list` before calling `sched->ops- > > > > >timedout_job()`. This > > > > means that when the job finally signals completion (e.g. in the > > > > IRQ > > > > handler), the scheduler won't call `sched->ops->free_job()`. As > > > > a > > > > result, > > > > the job and its resources won't be freed, leading to a memory > > > > leak. > > > > > > > > To resolve this issue, we create a new `drm_gpu_sched_stat` > > > > that > > > > allows a > > > > driver to skip the reset. This new status will indicate that > > > > the job > > > > should be reinserted into the pending list, and the driver will > > > > still > > > > signal its completion. > > > > > > > [...] > > > > > > diff --git a/include/drm/gpu_scheduler.h > > > > b/include/drm/gpu_scheduler.h > > > > index > > > > 1a7e377d4cbb4fc12ed93c548b236970217945e8..fe9043b6d43141bee831b > > > > 5fc16b927202a507d51 100644 > > > > --- a/include/drm/gpu_scheduler.h > > > > +++ b/include/drm/gpu_scheduler.h > > > > @@ -389,11 +389,13 @@ struct drm_sched_job { > > > > ?? * @DRM_GPU_SCHED_STAT_NONE: Reserved. Do not use. > > > > ?? * @DRM_GPU_SCHED_STAT_NOMINAL: Operation succeeded. > > > > ?? * @DRM_GPU_SCHED_STAT_ENODEV: Error: Device is not available > > > > anymore. > > > > + * @DRM_GPU_SCHED_STAT_RUNNING: GPU is still running, so skip > > > > the > > > > reset. > > > > > > s/GPU/job/ ? > > > > > > > ?? */ > > > > ? enum drm_gpu_sched_stat { > > > > ????? DRM_GPU_SCHED_STAT_NONE, > > > > ????? DRM_GPU_SCHED_STAT_NOMINAL, > > > > ????? DRM_GPU_SCHED_STAT_ENODEV, > > > > +??? DRM_GPU_SCHED_STAT_RUNNING, > > > > > > I am wondering if we could make it more obvious what is the > > > difference > > > between "nominal" and "running" and from whose point of view > > > should > > > those statuses be considered. > > ?> > So far we have "nominal" which means scheduler/hardware is > > working > > fine > > > but the job may or may have not been cancelled. With "running" we > > > kind > > > of split it into two sub-statuses and it would be nice for that > > > to be > > > intuitively visible from the naming. But I struggle to suggest an > > > elegant name while preserving nominal as is. > > > > I was thinking: how about changing DRM_GPU_SCHED_STAT_NOMINAL to > > DRM_GPU_SCHED_STAT_RESET (the hardware is fine, but we reset it)? > > > > Then, when we skip the reset, we would have > > DRM_GPU_SCHED_STAT_NOMINAL > > (which means the hardware is fine and we didn't reset it). > > > > I'm open to other suggestions. > > DRM_GPU_SCHED_STAT_RESET sounds like a good name and seems to paint a > consistent story between running - reset - enodev. > > > > Thinking out loud here - perhaps that is pointing towards an > > > alternative that instead of a new status, a new helper to re- > > > insert > > > the single job (like drm_sched_resubmit_job(sched, job)) would > > > fit > > > better? Although it would be more churn. > > > > > > > Although your solution might be more elegant, I'm worried that such > > a > > function could be used improperly by new users (e.g. being called > > in > > contexts other than `timedout_job()`). > > We could call it drm_sched_untimedout_job(). > > > I'd prefer to have a new status as it'll be use solely for > > `timedout_job()` (making it harder for users to use it > > inappropriately). > > With the addition of Matthew's feedback (calling > > `drm_sched_run_free_queue()` after adding the job to the pending > > list), > > I think it makes even more sense to keep it inside the timeout > > function. > > > > I hope others can chime in and give their opinions about your idea. > > Yeah - Philipp - Danilo - what do you prefer? Third enum with or a > new > helper? I'm also afraid that providing yet another helper for this specific case opens the door to abuse. We had (and still have) issues with the familiar drm_sched_resubmit_jobs() function. Christian has been very clear that this was a bad idea, and I'd rather not walk a road that looks similar to that one. I tend to think that the status codes are the appropriate mechanism to address this. They were, after all, invented to inform the scheduler about what is going on inside the driver. That said, currently, ENODEV is basically the only error, and everything unequal ENODEV (i.e., NOMINAL) is the "OK state". A timeout occurring and the GPU not hanging is, therefore, also "OK". Whatever the name will be, the docu for NOMINAL must also be adjusted. How about calling it "NORMAL" instead of "NOMINAL", since that state actually describes what is both OK and "the norm", i.e., most commonly the case? And I wouldn't call it RUNNING, since the GPU is also running in NOMINAL state. "NO_HANG" could hint more effectively at the fact that the GPU is, contrary to the scheduler's believe, not hanging. (I've been out for a few days and am catching up to a lot of things. Just had time to get deeper into this series. Apologies if my picture isn't complete yet) Thanks, P. > > Regards, > > Tvrtko > From phasta at mailbox.org Mon May 12 11:04:10 2025 From: phasta at mailbox.org (Philipp Stanner) Date: Mon, 12 May 2025 13:04:10 +0200 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> Message-ID: <82495d71bf97fbf6970ba6b016b3a121fc4b84f2.camel@mailbox.org> On Sat, 2025-05-03 at 17:59 -0300, Ma?ra Canal wrote: > When the DRM scheduler times out, it's possible that the GPU isn't > hung; > instead, a job may still be running, and there may be no valid reason > to > reset the hardware. This can occur in two situations: > > ? 1. The GPU exposes some mechanism that ensures the GPU is still > making > ???? progress. By checking this mechanism, we can safely skip the > reset, > ???? rearm the timeout, and allow the job to continue running until > ???? completion. This is the case for v3d and Etnaviv. Who is "we" and where is the reset skipped? In the timedout_job() callback? > ? 2. TDR has fired before the IRQ that signals the fence. Any concern about saying "Timeout" instead of "TDR"? I think most of us aren't familiar with that acronym. > Consequently, > ???? the job actually finishes, but it triggers a timeout before > signaling > ???? the completion fence. That formulation doesn't seem correct. Once the timeout fired, the job, as far as the GPU is concerned, is already finished, isn't it? What is the "completion fence"? In the scheduler, we call the fence returned by backend_ops.run_job() the "hardware fence". And who is the "it" in "it triggers a timeout"? I assume you want to say "the job has actually finished, but the scheduler triggers a timeout anyways". Also the purpose of that list is a bit unclear to me. It seems to be a list of problems, but point 1 seems valid? > > These two scenarios are problematic because we remove the job from > the > `sched->pending_list` before calling `sched->ops->timedout_job()`. Who is "we"? :) > This > means that when the job finally signals completion (e.g. in the IRQ > handler), A job doesn't signal completion. The hardware / driver signals job completion by signaling the hardware fence. > the scheduler won't call `sched->ops->free_job()`. As a result, > the job and its resources won't be freed, leading to a memory leak. OK, I think I get it. But isn't another explanation of the issue that the driver callback doesn't take care of cleaning up the job that has timed out (from the scheduler's perspective)? It's not clear to me that the scheduler actually contains a bug here, but rather is designed in a way that doesn't consider that some GPUs have special timeout requirements or, rather, can have bursts of slowness that don't actually indicate a timeout. I think the commit message should be very clear about whether this is an improvement of a design weakness or an actual bug fix. > > To resolve this issue, we create a new `drm_gpu_sched_stat` that > allows a > driver to skip the reset. This new status will indicate that the job > should be reinserted into the pending list, and the driver will still > signal its completion. Hmm, yes, I think that this is the right way to address that problem. +1 > > Signed-off-by: Ma?ra Canal > --- > ?drivers/gpu/drm/scheduler/sched_main.c | 14 ++++++++++++++ > ?include/drm/gpu_scheduler.h??????????? |? 2 ++ > ?2 files changed, 16 insertions(+) > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c > b/drivers/gpu/drm/scheduler/sched_main.c > index > 829579c41c6b5d8b2abce5ad373c7017469b7680..68ca827d77e32187a034309f881 > 135dbc639a9b4 100644 > --- a/drivers/gpu/drm/scheduler/sched_main.c > +++ b/drivers/gpu/drm/scheduler/sched_main.c > @@ -568,6 +568,17 @@ static void drm_sched_job_timedout(struct > work_struct *work) > ? job->sched->ops->free_job(job); > ? sched->free_guilty = false; > ? } > + > + /* > + * If the driver indicated that the GPU is still > running and wants to skip > + * the reset, reinsert the job back into the pending > list and realarm the > + * timeout. > + */ > + if (status == DRM_GPU_SCHED_STAT_RUNNING) { > + spin_lock(&sched->job_list_lock); > + list_add(&job->list, &sched->pending_list); > + spin_unlock(&sched->job_list_lock); > + } > ? } else { > ? spin_unlock(&sched->job_list_lock); > ? } > @@ -590,6 +601,9 @@ static void drm_sched_job_timedout(struct > work_struct *work) > ? * This function is typically used for reset recovery (see the docu > of > ? * drm_sched_backend_ops.timedout_job() for details). Do not call it > for > ? * scheduler teardown, i.e., before calling drm_sched_fini(). > + * > + * As it's used for reset recovery, drm_sched_stop() shouldn't be > called > + * if the scheduler skipped the timeout (DRM_SCHED_STAT_RUNNING). The same comment then applies to the counterpart, drm_sched_start(). We might also want to look into who uses drm_sched_wqueue_{start,stop} and consider if they need a comment. Though I don't expect you to do that. Those functions are hacky legacy anyways. P. > ? */ > ?void drm_sched_stop(struct drm_gpu_scheduler *sched, struct > drm_sched_job *bad) > ?{ > diff --git a/include/drm/gpu_scheduler.h > b/include/drm/gpu_scheduler.h > index > 1a7e377d4cbb4fc12ed93c548b236970217945e8..fe9043b6d43141bee831b5fc16b > 927202a507d51 100644 > --- a/include/drm/gpu_scheduler.h > +++ b/include/drm/gpu_scheduler.h > @@ -389,11 +389,13 @@ struct drm_sched_job { > ? * @DRM_GPU_SCHED_STAT_NONE: Reserved. Do not use. > ? * @DRM_GPU_SCHED_STAT_NOMINAL: Operation succeeded. > ? * @DRM_GPU_SCHED_STAT_ENODEV: Error: Device is not available > anymore. > + * @DRM_GPU_SCHED_STAT_RUNNING: GPU is still running, so skip the > reset. > ? */ > ?enum drm_gpu_sched_stat { > ? DRM_GPU_SCHED_STAT_NONE, > ? DRM_GPU_SCHED_STAT_NOMINAL, > ? DRM_GPU_SCHED_STAT_ENODEV, > + DRM_GPU_SCHED_STAT_RUNNING, > ?}; > ? > ?/** > From phasta at mailbox.org Mon May 12 11:13:39 2025 From: phasta at mailbox.org (Philipp Stanner) Date: Mon, 12 May 2025 13:13:39 +0200 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> Message-ID: On Tue, 2025-05-06 at 07:32 -0700, Matthew Brost wrote: > On Mon, May 05, 2025 at 07:41:09PM -0700, Matthew Brost wrote: > > On Sat, May 03, 2025 at 05:59:52PM -0300, Ma?ra Canal wrote: > > > When the DRM scheduler times out, it's possible that the GPU > > > isn't hung; > > > instead, a job may still be running, and there may be no valid > > > reason to > > > reset the hardware. This can occur in two situations: > > > > > > ? 1. The GPU exposes some mechanism that ensures the GPU is still > > > making > > > ???? progress. By checking this mechanism, we can safely skip the > > > reset, > > > ???? rearm the timeout, and allow the job to continue running > > > until > > > ???? completion. This is the case for v3d and Etnaviv. > > > ? 2. TDR has fired before the IRQ that signals the fence. > > > Consequently, > > > ???? the job actually finishes, but it triggers a timeout before > > > signaling > > > ???? the completion fence. > > > > > > > We have both of these cases in Xe too. We implement the requeuing > > in Xe > > via driver side function - xe_sched_add_pending_job but this looks > > better and will make use of this. > > > > > These two scenarios are problematic because we remove the job > > > from the > > > `sched->pending_list` before calling `sched->ops- > > > >timedout_job()`. This > > > means that when the job finally signals completion (e.g. in the > > > IRQ > > > handler), the scheduler won't call `sched->ops->free_job()`. As a > > > result, > > > the job and its resources won't be freed, leading to a memory > > > leak. > > > > > > To resolve this issue, we create a new `drm_gpu_sched_stat` that > > > allows a > > > driver to skip the reset. This new status will indicate that the > > > job > > > should be reinserted into the pending list, and the driver will > > > still > > > signal its completion. > > > > > > Signed-off-by: Ma?ra Canal > > > > Reviewed-by: Matthew Brost > > > > Wait - nevermind I think one issue is below. > > > > --- > > > ?drivers/gpu/drm/scheduler/sched_main.c | 14 ++++++++++++++ > > > ?include/drm/gpu_scheduler.h??????????? |? 2 ++ > > > ?2 files changed, 16 insertions(+) > > > > > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c > > > b/drivers/gpu/drm/scheduler/sched_main.c > > > index > > > 829579c41c6b5d8b2abce5ad373c7017469b7680..68ca827d77e32187a034309 > > > f881135dbc639a9b4 100644 > > > --- a/drivers/gpu/drm/scheduler/sched_main.c > > > +++ b/drivers/gpu/drm/scheduler/sched_main.c > > > @@ -568,6 +568,17 @@ static void drm_sched_job_timedout(struct > > > work_struct *work) > > > ? job->sched->ops->free_job(job); > > > ? sched->free_guilty = false; > > > ? } > > > + > > > + /* > > > + * If the driver indicated that the GPU is still > > > running and wants to skip > > > + * the reset, reinsert the job back into the > > > pending list and realarm the > > > + * timeout. > > > + */ > > > + if (status == DRM_GPU_SCHED_STAT_RUNNING) { > > > + spin_lock(&sched->job_list_lock); > > > + list_add(&job->list, &sched- > > > >pending_list); > > > + spin_unlock(&sched->job_list_lock); > > > + } > > I think you need to requeue free_job wq here. It is possible the > free_job wq ran, didn't find a job, goes to sleep, then we add a > signaled job here which will never get freed. I wonder if that could be solved by holding job_list_lock a bit longer. free_job_work will try to check the list for the next signaled job, but will wait for the lock. If that works, we could completely rely on the standard mechanism without requeuing, which would be neat. P. > > Matt > > > > ? } else { > > > ? spin_unlock(&sched->job_list_lock); > > > ? } > > > @@ -590,6 +601,9 @@ static void drm_sched_job_timedout(struct > > > work_struct *work) > > > ? * This function is typically used for reset recovery (see the > > > docu of > > > ? * drm_sched_backend_ops.timedout_job() for details). Do not > > > call it for > > > ? * scheduler teardown, i.e., before calling drm_sched_fini(). > > > + * > > > + * As it's used for reset recovery, drm_sched_stop() shouldn't > > > be called > > > + * if the scheduler skipped the timeout > > > (DRM_SCHED_STAT_RUNNING). > > > ? */ > > > ?void drm_sched_stop(struct drm_gpu_scheduler *sched, struct > > > drm_sched_job *bad) > > > ?{ > > > diff --git a/include/drm/gpu_scheduler.h > > > b/include/drm/gpu_scheduler.h > > > index > > > 1a7e377d4cbb4fc12ed93c548b236970217945e8..fe9043b6d43141bee831b5f > > > c16b927202a507d51 100644 > > > --- a/include/drm/gpu_scheduler.h > > > +++ b/include/drm/gpu_scheduler.h > > > @@ -389,11 +389,13 @@ struct drm_sched_job { > > > ? * @DRM_GPU_SCHED_STAT_NONE: Reserved. Do not use. > > > ? * @DRM_GPU_SCHED_STAT_NOMINAL: Operation succeeded. > > > ? * @DRM_GPU_SCHED_STAT_ENODEV: Error: Device is not available > > > anymore. > > > + * @DRM_GPU_SCHED_STAT_RUNNING: GPU is still running, so skip > > > the reset. > > > ? */ > > > ?enum drm_gpu_sched_stat { > > > ? DRM_GPU_SCHED_STAT_NONE, > > > ? DRM_GPU_SCHED_STAT_NOMINAL, > > > ? DRM_GPU_SCHED_STAT_ENODEV, > > > + DRM_GPU_SCHED_STAT_RUNNING, > > > ?}; > > > ? > > > ?/** > > > > > > -- > > > 2.49.0 > > > From mcanal at igalia.com Mon May 12 13:59:33 2025 From: mcanal at igalia.com (=?UTF-8?Q?Ma=C3=ADra_Canal?=) Date: Mon, 12 May 2025 10:59:33 -0300 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: <82495d71bf97fbf6970ba6b016b3a121fc4b84f2.camel@mailbox.org> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> <82495d71bf97fbf6970ba6b016b3a121fc4b84f2.camel@mailbox.org> Message-ID: Hi Philipp, On 12/05/25 08:04, Philipp Stanner wrote: > On Sat, 2025-05-03 at 17:59 -0300, Ma?ra Canal wrote: >> When the DRM scheduler times out, it's possible that the GPU isn't >> hung; >> instead, a job may still be running, and there may be no valid reason >> to >> reset the hardware. This can occur in two situations: >> >> ? 1. The GPU exposes some mechanism that ensures the GPU is still >> making >> ???? progress. By checking this mechanism, we can safely skip the >> reset, >> ???? rearm the timeout, and allow the job to continue running until >> ???? completion. This is the case for v3d and Etnaviv. > > Who is "we" and where is the reset skipped? In the timedout_job() > callback? "we" is just a generic way to say "we, the driver developers". The reset is skipped in the `timedout_job()` callback. > >> ? 2. TDR has fired before the IRQ that signals the fence. > > Any concern about saying "Timeout" instead of "TDR"? I think most of us > aren't familiar with that acronym. I'll use "Timeout" in v2. > >> Consequently, >> ???? the job actually finishes, but it triggers a timeout before >> signaling >> ???? the completion fence. > > That formulation doesn't seem correct. Once the timeout fired, the job, > as far as the GPU is concerned, is already finished, isn't it? Yes, but there is a race-condition between the IRQ handler (which will indeed signal the fence) and timeout handler. > > What is the "completion fence"? In the scheduler, we call the fence > returned by backend_ops.run_job() the "hardware fence". > > And who is the "it" in "it triggers a timeout"? I assume you want to > say "the job has actually finished, but the scheduler triggers a > timeout anyways". > I'll address it in v2. > > Also the purpose of that list is a bit unclear to me. It seems to be a > list of problems, but point 1 seems valid? > Point 2 is also valid, otherwise Xe, Panfrost, Lima, and Etnaviv wouldn't have to handle this situation. TDR can "win" the race-condition and although the hardware fence will be signaled in the IRQ handler, the job won't be freed. > >> >> These two scenarios are problematic because we remove the job from >> the >> `sched->pending_list` before calling `sched->ops->timedout_job()`. > > Who is "we"? :) Ditto. > > >> This >> means that when the job finally signals completion (e.g. in the IRQ >> handler), > > A job doesn't signal completion. > > The hardware / driver signals job completion by signaling the hardware > fence. I'll address it in v2. > >> the scheduler won't call `sched->ops->free_job()`. As a result, >> the job and its resources won't be freed, leading to a memory leak. > > OK, I think I get it. But isn't another explanation of the issue that > the driver callback doesn't take care of cleaning up the job that has > timed out (from the scheduler's perspective)? > > It's not clear to me that the scheduler actually contains a bug here, > but rather is designed in a way that doesn't consider that some GPUs > have special timeout requirements or, rather, can have bursts of > slowness that don't actually indicate a timeout. > > I think the commit message should be very clear about whether this is > an improvement of a design weakness or an actual bug fix. > >> >> To resolve this issue, we create a new `drm_gpu_sched_stat` that >> allows a >> driver to skip the reset. This new status will indicate that the job >> should be reinserted into the pending list, and the driver will still >> signal its completion. > > Hmm, yes, I think that this is the right way to address that problem. > +1 > > >> >> Signed-off-by: Ma?ra Canal >> --- >> ?drivers/gpu/drm/scheduler/sched_main.c | 14 ++++++++++++++ >> ?include/drm/gpu_scheduler.h??????????? |? 2 ++ >> ?2 files changed, 16 insertions(+) >> >> diff --git a/drivers/gpu/drm/scheduler/sched_main.c >> b/drivers/gpu/drm/scheduler/sched_main.c >> index >> 829579c41c6b5d8b2abce5ad373c7017469b7680..68ca827d77e32187a034309f881 >> 135dbc639a9b4 100644 >> --- a/drivers/gpu/drm/scheduler/sched_main.c >> +++ b/drivers/gpu/drm/scheduler/sched_main.c >> @@ -568,6 +568,17 @@ static void drm_sched_job_timedout(struct >> work_struct *work) >> ? job->sched->ops->free_job(job); >> ? sched->free_guilty = false; >> ? } >> + >> + /* >> + * If the driver indicated that the GPU is still >> running and wants to skip >> + * the reset, reinsert the job back into the pending >> list and realarm the >> + * timeout. >> + */ >> + if (status == DRM_GPU_SCHED_STAT_RUNNING) { >> + spin_lock(&sched->job_list_lock); >> + list_add(&job->list, &sched->pending_list); >> + spin_unlock(&sched->job_list_lock); >> + } >> ? } else { >> ? spin_unlock(&sched->job_list_lock); >> ? } >> @@ -590,6 +601,9 @@ static void drm_sched_job_timedout(struct >> work_struct *work) >> ? * This function is typically used for reset recovery (see the docu >> of >> ? * drm_sched_backend_ops.timedout_job() for details). Do not call it >> for >> ? * scheduler teardown, i.e., before calling drm_sched_fini(). >> + * >> + * As it's used for reset recovery, drm_sched_stop() shouldn't be >> called >> + * if the scheduler skipped the timeout (DRM_SCHED_STAT_RUNNING). > > The same comment then applies to the counterpart, drm_sched_start(). > Thanks for your review! I'll address all your comments in v2. Best Regards, - Ma?ra > We might also want to look into who uses drm_sched_wqueue_{start,stop} > and consider if they need a comment. Though I don't expect you to do > that. Those functions are hacky legacy anyways. > > > P. > >> ? */ >> ?void drm_sched_stop(struct drm_gpu_scheduler *sched, struct >> drm_sched_job *bad) >> ?{ >> diff --git a/include/drm/gpu_scheduler.h >> b/include/drm/gpu_scheduler.h >> index >> 1a7e377d4cbb4fc12ed93c548b236970217945e8..fe9043b6d43141bee831b5fc16b >> 927202a507d51 100644 >> --- a/include/drm/gpu_scheduler.h >> +++ b/include/drm/gpu_scheduler.h >> @@ -389,11 +389,13 @@ struct drm_sched_job { >> ? * @DRM_GPU_SCHED_STAT_NONE: Reserved. Do not use. >> ? * @DRM_GPU_SCHED_STAT_NOMINAL: Operation succeeded. >> ? * @DRM_GPU_SCHED_STAT_ENODEV: Error: Device is not available >> anymore. >> + * @DRM_GPU_SCHED_STAT_RUNNING: GPU is still running, so skip the >> reset. >> ? */ >> ?enum drm_gpu_sched_stat { >> ? DRM_GPU_SCHED_STAT_NONE, >> ? DRM_GPU_SCHED_STAT_NOMINAL, >> ? DRM_GPU_SCHED_STAT_ENODEV, >> + DRM_GPU_SCHED_STAT_RUNNING, >> ?}; >> >> ?/** >> > From mcanal at igalia.com Mon May 12 14:04:50 2025 From: mcanal at igalia.com (=?UTF-8?Q?Ma=C3=ADra_Canal?=) Date: Mon, 12 May 2025 11:04:50 -0300 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> Message-ID: <22fdf8aa-437a-4d28-886a-fe10e957edfa@igalia.com> Hi Philipp, On 12/05/25 08:13, Philipp Stanner wrote: > On Tue, 2025-05-06 at 07:32 -0700, Matthew Brost wrote: >> On Mon, May 05, 2025 at 07:41:09PM -0700, Matthew Brost wrote: >>> On Sat, May 03, 2025 at 05:59:52PM -0300, Ma?ra Canal wrote: >>>> When the DRM scheduler times out, it's possible that the GPU >>>> isn't hung; >>>> instead, a job may still be running, and there may be no valid >>>> reason to >>>> reset the hardware. This can occur in two situations: >>>> >>>> ? 1. The GPU exposes some mechanism that ensures the GPU is still >>>> making >>>> ???? progress. By checking this mechanism, we can safely skip the >>>> reset, >>>> ???? rearm the timeout, and allow the job to continue running >>>> until >>>> ???? completion. This is the case for v3d and Etnaviv. >>>> ? 2. TDR has fired before the IRQ that signals the fence. >>>> Consequently, >>>> ???? the job actually finishes, but it triggers a timeout before >>>> signaling >>>> ???? the completion fence. >>>> >>> >>> We have both of these cases in Xe too. We implement the requeuing >>> in Xe >>> via driver side function - xe_sched_add_pending_job but this looks >>> better and will make use of this. >>> >>>> These two scenarios are problematic because we remove the job >>>> from the >>>> `sched->pending_list` before calling `sched->ops- >>>>> timedout_job()`. This >>>> means that when the job finally signals completion (e.g. in the >>>> IRQ >>>> handler), the scheduler won't call `sched->ops->free_job()`. As a >>>> result, >>>> the job and its resources won't be freed, leading to a memory >>>> leak. >>>> >>>> To resolve this issue, we create a new `drm_gpu_sched_stat` that >>>> allows a >>>> driver to skip the reset. This new status will indicate that the >>>> job >>>> should be reinserted into the pending list, and the driver will >>>> still >>>> signal its completion. >>>> >>>> Signed-off-by: Ma?ra Canal >>> >>> Reviewed-by: Matthew Brost >>> >> >> Wait - nevermind I think one issue is below. >> >>>> --- >>>> ?drivers/gpu/drm/scheduler/sched_main.c | 14 ++++++++++++++ >>>> ?include/drm/gpu_scheduler.h??????????? |? 2 ++ >>>> ?2 files changed, 16 insertions(+) >>>> >>>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c >>>> b/drivers/gpu/drm/scheduler/sched_main.c >>>> index >>>> 829579c41c6b5d8b2abce5ad373c7017469b7680..68ca827d77e32187a034309 >>>> f881135dbc639a9b4 100644 >>>> --- a/drivers/gpu/drm/scheduler/sched_main.c >>>> +++ b/drivers/gpu/drm/scheduler/sched_main.c >>>> @@ -568,6 +568,17 @@ static void drm_sched_job_timedout(struct >>>> work_struct *work) >>>> ? job->sched->ops->free_job(job); >>>> ? sched->free_guilty = false; >>>> ? } >>>> + >>>> + /* >>>> + * If the driver indicated that the GPU is still >>>> running and wants to skip >>>> + * the reset, reinsert the job back into the >>>> pending list and realarm the >>>> + * timeout. >>>> + */ >>>> + if (status == DRM_GPU_SCHED_STAT_RUNNING) { >>>> + spin_lock(&sched->job_list_lock); >>>> + list_add(&job->list, &sched- >>>>> pending_list); >>>> + spin_unlock(&sched->job_list_lock); >>>> + } >> >> I think you need to requeue free_job wq here. It is possible the >> free_job wq ran, didn't find a job, goes to sleep, then we add a >> signaled job here which will never get freed. > > I wonder if that could be solved by holding job_list_lock a bit longer. > free_job_work will try to check the list for the next signaled job, but > will wait for the lock. > > If that works, we could completely rely on the standard mechanism > without requeuing, which would be neat. I believe it works. However, the tradeoff would be holding the lock for the entire reset of the GPU (in the cases the GPU actually hanged), which looks like a lot of time. Do you think it's reasonable to do so? Best Regards, - Ma?ra > > P. > >> >> Matt >> From phasta at mailbox.org Mon May 12 14:09:21 2025 From: phasta at mailbox.org (Philipp Stanner) Date: Mon, 12 May 2025 16:09:21 +0200 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: <22fdf8aa-437a-4d28-886a-fe10e957edfa@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> <22fdf8aa-437a-4d28-886a-fe10e957edfa@igalia.com> Message-ID: <50debbe5a4856a744b8ac89048405be89f936a6a.camel@mailbox.org> On Mon, 2025-05-12 at 11:04 -0300, Ma?ra Canal wrote: > Hi Philipp, > > On 12/05/25 08:13, Philipp Stanner wrote: > > On Tue, 2025-05-06 at 07:32 -0700, Matthew Brost wrote: > > > On Mon, May 05, 2025 at 07:41:09PM -0700, Matthew Brost wrote: > > > > On Sat, May 03, 2025 at 05:59:52PM -0300, Ma?ra Canal wrote: > > > > > When the DRM scheduler times out, it's possible that the GPU > > > > > isn't hung; > > > > > instead, a job may still be running, and there may be no > > > > > valid > > > > > reason to > > > > > reset the hardware. This can occur in two situations: > > > > > > > > > > ?? 1. The GPU exposes some mechanism that ensures the GPU is > > > > > still > > > > > making > > > > > ????? progress. By checking this mechanism, we can safely > > > > > skip the > > > > > reset, > > > > > ????? rearm the timeout, and allow the job to continue > > > > > running > > > > > until > > > > > ????? completion. This is the case for v3d and Etnaviv. > > > > > ?? 2. TDR has fired before the IRQ that signals the fence. > > > > > Consequently, > > > > > ????? the job actually finishes, but it triggers a timeout > > > > > before > > > > > signaling > > > > > ????? the completion fence. > > > > > > > > > > > > > We have both of these cases in Xe too. We implement the > > > > requeuing > > > > in Xe > > > > via driver side function - xe_sched_add_pending_job but this > > > > looks > > > > better and will make use of this. > > > > > > > > > These two scenarios are problematic because we remove the job > > > > > from the > > > > > `sched->pending_list` before calling `sched->ops- > > > > > > timedout_job()`. This > > > > > means that when the job finally signals completion (e.g. in > > > > > the > > > > > IRQ > > > > > handler), the scheduler won't call `sched->ops->free_job()`. > > > > > As a > > > > > result, > > > > > the job and its resources won't be freed, leading to a memory > > > > > leak. > > > > > > > > > > To resolve this issue, we create a new `drm_gpu_sched_stat` > > > > > that > > > > > allows a > > > > > driver to skip the reset. This new status will indicate that > > > > > the > > > > > job > > > > > should be reinserted into the pending list, and the driver > > > > > will > > > > > still > > > > > signal its completion. > > > > > > > > > > Signed-off-by: Ma?ra Canal > > > > > > > > Reviewed-by: Matthew Brost > > > > > > > > > > Wait - nevermind I think one issue is below. > > > > > > > > --- > > > > > ??drivers/gpu/drm/scheduler/sched_main.c | 14 ++++++++++++++ > > > > > ??include/drm/gpu_scheduler.h??????????? |? 2 ++ > > > > > ??2 files changed, 16 insertions(+) > > > > > > > > > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c > > > > > b/drivers/gpu/drm/scheduler/sched_main.c > > > > > index > > > > > 829579c41c6b5d8b2abce5ad373c7017469b7680..68ca827d77e32187a03 > > > > > 4309 > > > > > f881135dbc639a9b4 100644 > > > > > --- a/drivers/gpu/drm/scheduler/sched_main.c > > > > > +++ b/drivers/gpu/drm/scheduler/sched_main.c > > > > > @@ -568,6 +568,17 @@ static void > > > > > drm_sched_job_timedout(struct > > > > > work_struct *work) > > > > > ?? job->sched->ops->free_job(job); > > > > > ?? sched->free_guilty = false; > > > > > ?? } > > > > > + > > > > > + /* > > > > > + * If the driver indicated that the GPU is > > > > > still > > > > > running and wants to skip > > > > > + * the reset, reinsert the job back into the > > > > > pending list and realarm the > > > > > + * timeout. > > > > > + */ > > > > > + if (status == DRM_GPU_SCHED_STAT_RUNNING) { > > > > > + spin_lock(&sched->job_list_lock); > > > > > + list_add(&job->list, &sched- > > > > > > pending_list); > > > > > + spin_unlock(&sched->job_list_lock); > > > > > + } > > > > > > I think you need to requeue free_job wq here. It is possible the > > > free_job wq ran, didn't find a job, goes to sleep, then we add a > > > signaled job here which will never get freed. > > > > I wonder if that could be solved by holding job_list_lock a bit > > longer. > > free_job_work will try to check the list for the next signaled job, > > but > > will wait for the lock. > > > > If that works, we could completely rely on the standard mechanism > > without requeuing, which would be neat. > > I believe it works. However, the tradeoff would be holding the lock > for > the entire reset of the GPU (in the cases the GPU actually hanged), > which looks like a lot of time. > > Do you think it's reasonable to do so? The scheduler only has three distinct work items, run_job, free_job and timeout. timeout runs only serially, so that's not relevant; and run_job() and free_job() should be halted in the timeout handler through drm_sched_stop() anyways. Moreover, timeouts should be rare events. So I'd say yes, the clarity of the code trumps here. Cheers, P. > > Best Regards, > - Ma?ra > > > > > P. > > > > > > > > Matt > > > > > From phasta at mailbox.org Mon May 12 14:58:56 2025 From: phasta at mailbox.org (Philipp Stanner) Date: Mon, 12 May 2025 16:58:56 +0200 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: <50debbe5a4856a744b8ac89048405be89f936a6a.camel@mailbox.org> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> <22fdf8aa-437a-4d28-886a-fe10e957edfa@igalia.com> <50debbe5a4856a744b8ac89048405be89f936a6a.camel@mailbox.org> Message-ID: On Mon, 2025-05-12 at 16:09 +0200, Philipp Stanner wrote: > On Mon, 2025-05-12 at 11:04 -0300, Ma?ra Canal wrote: > > Hi Philipp, > > > > On 12/05/25 08:13, Philipp Stanner wrote: > > > On Tue, 2025-05-06 at 07:32 -0700, Matthew Brost wrote: > > > > On Mon, May 05, 2025 at 07:41:09PM -0700, Matthew Brost wrote: > > > > > On Sat, May 03, 2025 at 05:59:52PM -0300, Ma?ra Canal wrote: > > > > > > When the DRM scheduler times out, it's possible that the > > > > > > GPU > > > > > > isn't hung; > > > > > > instead, a job may still be running, and there may be no > > > > > > valid > > > > > > reason to > > > > > > reset the hardware. This can occur in two situations: > > > > > > > > > > > > ?? 1. The GPU exposes some mechanism that ensures the GPU > > > > > > is > > > > > > still > > > > > > making > > > > > > ????? progress. By checking this mechanism, we can safely > > > > > > skip the > > > > > > reset, > > > > > > ????? rearm the timeout, and allow the job to continue > > > > > > running > > > > > > until > > > > > > ????? completion. This is the case for v3d and Etnaviv. > > > > > > ?? 2. TDR has fired before the IRQ that signals the fence. > > > > > > Consequently, > > > > > > ????? the job actually finishes, but it triggers a timeout > > > > > > before > > > > > > signaling > > > > > > ????? the completion fence. > > > > > > > > > > > > > > > > We have both of these cases in Xe too. We implement the > > > > > requeuing > > > > > in Xe > > > > > via driver side function - xe_sched_add_pending_job but this > > > > > looks > > > > > better and will make use of this. > > > > > > > > > > > These two scenarios are problematic because we remove the > > > > > > job > > > > > > from the > > > > > > `sched->pending_list` before calling `sched->ops- > > > > > > > timedout_job()`. This > > > > > > means that when the job finally signals completion (e.g. in > > > > > > the > > > > > > IRQ > > > > > > handler), the scheduler won't call `sched->ops- > > > > > > >free_job()`. > > > > > > As a > > > > > > result, > > > > > > the job and its resources won't be freed, leading to a > > > > > > memory > > > > > > leak. > > > > > > > > > > > > To resolve this issue, we create a new `drm_gpu_sched_stat` > > > > > > that > > > > > > allows a > > > > > > driver to skip the reset. This new status will indicate > > > > > > that > > > > > > the > > > > > > job > > > > > > should be reinserted into the pending list, and the driver > > > > > > will > > > > > > still > > > > > > signal its completion. > > > > > > > > > > > > Signed-off-by: Ma?ra Canal > > > > > > > > > > Reviewed-by: Matthew Brost > > > > > > > > > > > > > Wait - nevermind I think one issue is below. > > > > > > > > > > --- > > > > > > ??drivers/gpu/drm/scheduler/sched_main.c | 14 > > > > > > ++++++++++++++ > > > > > > ??include/drm/gpu_scheduler.h??????????? |? 2 ++ > > > > > > ??2 files changed, 16 insertions(+) > > > > > > > > > > > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c > > > > > > b/drivers/gpu/drm/scheduler/sched_main.c > > > > > > index > > > > > > 829579c41c6b5d8b2abce5ad373c7017469b7680..68ca827d77e32187a > > > > > > 03 > > > > > > 4309 > > > > > > f881135dbc639a9b4 100644 > > > > > > --- a/drivers/gpu/drm/scheduler/sched_main.c > > > > > > +++ b/drivers/gpu/drm/scheduler/sched_main.c > > > > > > @@ -568,6 +568,17 @@ static void > > > > > > drm_sched_job_timedout(struct > > > > > > work_struct *work) > > > > > > ?? job->sched->ops->free_job(job); > > > > > > ?? sched->free_guilty = false; > > > > > > ?? } > > > > > > + > > > > > > + /* > > > > > > + * If the driver indicated that the GPU is > > > > > > still > > > > > > running and wants to skip > > > > > > + * the reset, reinsert the job back into > > > > > > the > > > > > > pending list and realarm the > > > > > > + * timeout. > > > > > > + */ > > > > > > + if (status == DRM_GPU_SCHED_STAT_RUNNING) > > > > > > { > > > > > > + spin_lock(&sched->job_list_lock); > > > > > > + list_add(&job->list, &sched- > > > > > > > pending_list); > > > > > > + spin_unlock(&sched- > > > > > > >job_list_lock); > > > > > > + } > > > > > > > > I think you need to requeue free_job wq here. It is possible > > > > the > > > > free_job wq ran, didn't find a job, goes to sleep, then we add > > > > a > > > > signaled job here which will never get freed. > > > > > > I wonder if that could be solved by holding job_list_lock a bit > > > longer. > > > free_job_work will try to check the list for the next signaled > > > job, > > > but > > > will wait for the lock. > > > > > > If that works, we could completely rely on the standard mechanism > > > without requeuing, which would be neat. > > > > I believe it works. However, the tradeoff would be holding the lock > > for > > the entire reset of the GPU (in the cases the GPU actually hanged), > > which looks like a lot of time. > > > > Do you think it's reasonable to do so? > > The scheduler only has three distinct work items, run_job, free_job > and > timeout. > > timeout runs only serially, so that's not relevant; and run_job() and > free_job() should be halted in the timeout handler through > drm_sched_stop() anyways. > > Moreover, timeouts should be rare events. > > So I'd say yes, the clarity of the code trumps here. Forget about that, turns out to be an insane idea. Thx to Danilo for the feedback. Many drivers take locks in timedout_job(), for example pvr_queue_timedout_job(). So that's insane deadlock danger. Not good. So it seems the only option left is Matt's idea? But that certainly needs an explanatory comment, too. P. > > Cheers, > P. > > > > > > Best Regards, > > - Ma?ra > > > > > > > > P. > > > > > > > > > > > Matt > > > > > > > > > From phasta at mailbox.org Tue May 13 07:26:26 2025 From: phasta at mailbox.org (Philipp Stanner) Date: Tue, 13 May 2025 09:26:26 +0200 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> Message-ID: <4242fd242c7e16d0ecdf11c5d0ad795efda727a5.camel@mailbox.org> On Sat, 2025-05-03 at 17:59 -0300, Ma?ra Canal wrote: > When the DRM scheduler times out, it's possible that the GPU isn't > hung; > instead, a job may still be running, and there may be no valid reason > to > reset the hardware. This can occur in two situations: > > ? 1. The GPU exposes some mechanism that ensures the GPU is still > making > ???? progress. By checking this mechanism, we can safely skip the > reset, > ???? rearm the timeout, and allow the job to continue running until > ???? completion. This is the case for v3d and Etnaviv. > ? 2. TDR has fired before the IRQ that signals the fence. > Consequently, > ???? the job actually finishes, but it triggers a timeout before > signaling > ???? the completion fence. > > These two scenarios are problematic because we remove the job from > the > `sched->pending_list` before calling `sched->ops->timedout_job()`. > This > means that when the job finally signals completion (e.g. in the IRQ > handler), the scheduler won't call `sched->ops->free_job()`. As a > result, > the job and its resources won't be freed, leading to a memory leak. We have discussed this and discovered another, related issue. See below. > > To resolve this issue, we create a new `drm_gpu_sched_stat` that > allows a > driver to skip the reset. This new status will indicate that the job > should be reinserted into the pending list, and the driver will still > signal its completion. > > Signed-off-by: Ma?ra Canal > --- > ?drivers/gpu/drm/scheduler/sched_main.c | 14 ++++++++++++++ > ?include/drm/gpu_scheduler.h??????????? |? 2 ++ > ?2 files changed, 16 insertions(+) > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c > b/drivers/gpu/drm/scheduler/sched_main.c > index > 829579c41c6b5d8b2abce5ad373c7017469b7680..68ca827d77e32187a034309f881 > 135dbc639a9b4 100644 > --- a/drivers/gpu/drm/scheduler/sched_main.c > +++ b/drivers/gpu/drm/scheduler/sched_main.c > @@ -568,6 +568,17 @@ static void drm_sched_job_timedout(struct > work_struct *work) So, the fundamental design problem we have is that the scheduler assumes that when a timeout occurs, the GPU is completely hung. Your patch addresses another aspect of that very problem. But if the GPU is not hung, it can signal the hardware fence at any moment. So that's racy. It could, theoretically, lead to backend_ops.timedout_job() being called with a signaled job, i.e., a job that is not really timed out. Would you say this is *the same* issue you're describing, or a separate one? It seems to me that it's a separate one. Anyways. What I propose is that we wait until your series here has been merged. Once that's done, we should document that drivers should expect that backend_ops.timedout_job() can get called with a job that has not actually timed out, and tell the scheduler about it through DRM_GPU_SCHED_STAT_NOT_HANGING. Then the scheduler reverts the timeout's actions, as you propose here. > ? job->sched->ops->free_job(job); > ? sched->free_guilty = false; > ? } > + > + /* > + * If the driver indicated that the GPU is still > running and wants to skip > + * the reset, reinsert the job back into the pending > list and realarm the > + * timeout. > + */ > + if (status == DRM_GPU_SCHED_STAT_RUNNING) { > + spin_lock(&sched->job_list_lock); > + list_add(&job->list, &sched->pending_list); > + spin_unlock(&sched->job_list_lock); > + } btw, if you go for Matt's requeue work item approach, it'll be better to write a helper function with a clear name for all that. drm_sched_job_reinsert_on_false_timout() maybe. P. > ? } else { > ? spin_unlock(&sched->job_list_lock); > ? } > @@ -590,6 +601,9 @@ static void drm_sched_job_timedout(struct > work_struct *work) > ? * This function is typically used for reset recovery (see the docu > of > ? * drm_sched_backend_ops.timedout_job() for details). Do not call it > for > ? * scheduler teardown, i.e., before calling drm_sched_fini(). > + * > + * As it's used for reset recovery, drm_sched_stop() shouldn't be > called > + * if the scheduler skipped the timeout (DRM_SCHED_STAT_RUNNING). > ? */ > ?void drm_sched_stop(struct drm_gpu_scheduler *sched, struct > drm_sched_job *bad) > ?{ > diff --git a/include/drm/gpu_scheduler.h > b/include/drm/gpu_scheduler.h > index > 1a7e377d4cbb4fc12ed93c548b236970217945e8..fe9043b6d43141bee831b5fc16b > 927202a507d51 100644 > --- a/include/drm/gpu_scheduler.h > +++ b/include/drm/gpu_scheduler.h > @@ -389,11 +389,13 @@ struct drm_sched_job { > ? * @DRM_GPU_SCHED_STAT_NONE: Reserved. Do not use. > ? * @DRM_GPU_SCHED_STAT_NOMINAL: Operation succeeded. > ? * @DRM_GPU_SCHED_STAT_ENODEV: Error: Device is not available > anymore. > + * @DRM_GPU_SCHED_STAT_RUNNING: GPU is still running, so skip the > reset. > ? */ > ?enum drm_gpu_sched_stat { > ? DRM_GPU_SCHED_STAT_NONE, > ? DRM_GPU_SCHED_STAT_NOMINAL, > ? DRM_GPU_SCHED_STAT_ENODEV, > + DRM_GPU_SCHED_STAT_RUNNING, > ?}; > ? > ?/** > From pierre-eric at damsy.net Wed May 14 12:25:54 2025 From: pierre-eric at damsy.net (Pierre-Eric Pelloux-Prayer) Date: Wed, 14 May 2025 14:25:54 +0200 Subject: [PATCH v9 00/10] Improve gpu_scheduler trace events + UAPI In-Reply-To: <20250424083834.15518-1-pierre-eric.pelloux-prayer@amd.com> References: <20250424083834.15518-1-pierre-eric.pelloux-prayer@amd.com> Message-ID: <34321866-6991-4aa0-98e9-6d6b7c37e8e4@damsy.net> Hi Philipp, Did you get a chance to take a look at the latest revision of this series? Thanks, Pierre-Eric Le 24/04/2025 ? 10:38, Pierre-Eric Pelloux-Prayer a ?crit?: > Hi, > > The initial goal of this series was to improve the drm and amdgpu > trace events to be able to expose more of the inner workings of > the scheduler and drivers to developers via tools. > > Then, the series evolved to become focused only on gpu_scheduler. > The changes around vblank events will be part of a different > series, as well as the amdgpu ones. > > Moreover Sima suggested to make some trace events stable uAPI, > so tools can rely on them long term. > > The first patches extend and cleanup the gpu scheduler events, > then add a documentation entry in drm-uapi.rst. > > The last 2 patches are new in v8. One is based on a suggestion > from Tvrtko and gets rid of drm_sched_job::id. The other is a > cleanup of amdgpu trace events to use the fence=%llu:%llu format. > > The drm_sched_job patches don't affect gpuvis which has code to parse > the gpu_scheduler events but these events are not enabled. > > Changes since v8: > * swapped patches 8 & 9 > * rebased on drm-next > > Changes since v7: > * uint64_t -> u64 > * reworked dependencies tracing (Tvrtko) > * use common name prefix for all events (Tvrtko) > * dropped drm_sched_job::id (Tvrtko) > > Useful links: > - userspace tool using the updated events: > https://gitlab.freedesktop.org/tomstdenis/umr/-/merge_requests/37 > - v8: > https://lists.freedesktop.org/archives/dri-devel/2025-March/496781.html > > Pierre-Eric Pelloux-Prayer (10): > drm/debugfs: output client_id in in drm_clients_info > drm/sched: store the drm client_id in drm_sched_fence > drm/sched: add device name to the drm_sched_process_job event > drm/sched: cleanup gpu_scheduler trace events > drm/sched: trace dependencies for gpu jobs > drm/sched: add the drm_client_id to the drm_sched_run/exec_job events > drm/sched: cleanup event names > drm: get rid of drm_sched_job::id > drm/doc: document some tracepoints as uAPI > drm/amdgpu: update trace format to match gpu_scheduler_trace > > Documentation/gpu/drm-uapi.rst | 19 ++++ > drivers/accel/amdxdna/aie2_ctx.c | 3 +- > drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 2 +- > drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 3 +- > drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 8 +- > drivers/gpu/drm/amd/amdgpu/amdgpu_job.h | 3 +- > drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h | 32 +++--- > drivers/gpu/drm/drm_debugfs.c | 10 +- > drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c | 2 +- > drivers/gpu/drm/imagination/pvr_job.c | 2 +- > drivers/gpu/drm/imagination/pvr_queue.c | 5 +- > drivers/gpu/drm/imagination/pvr_queue.h | 2 +- > drivers/gpu/drm/lima/lima_gem.c | 2 +- > drivers/gpu/drm/lima/lima_sched.c | 6 +- > drivers/gpu/drm/lima/lima_sched.h | 3 +- > drivers/gpu/drm/msm/msm_gem_submit.c | 8 +- > drivers/gpu/drm/nouveau/nouveau_sched.c | 3 +- > drivers/gpu/drm/panfrost/panfrost_drv.c | 2 +- > drivers/gpu/drm/panthor/panthor_drv.c | 3 +- > drivers/gpu/drm/panthor/panthor_mmu.c | 2 +- > drivers/gpu/drm/panthor/panthor_sched.c | 5 +- > drivers/gpu/drm/panthor/panthor_sched.h | 3 +- > .../gpu/drm/scheduler/gpu_scheduler_trace.h | 100 +++++++++++++----- > drivers/gpu/drm/scheduler/sched_entity.c | 16 ++- > drivers/gpu/drm/scheduler/sched_fence.c | 4 +- > drivers/gpu/drm/scheduler/sched_internal.h | 2 +- > drivers/gpu/drm/scheduler/sched_main.c | 11 +- > .../gpu/drm/scheduler/tests/mock_scheduler.c | 2 +- > drivers/gpu/drm/v3d/v3d_submit.c | 2 +- > drivers/gpu/drm/xe/xe_sched_job.c | 3 +- > include/drm/gpu_scheduler.h | 13 ++- > 31 files changed, 184 insertions(+), 97 deletions(-) > From phasta at mailbox.org Wed May 14 12:44:14 2025 From: phasta at mailbox.org (Philipp Stanner) Date: Wed, 14 May 2025 14:44:14 +0200 Subject: [PATCH v9 02/10] drm/sched: store the drm client_id in drm_sched_fence In-Reply-To: <20250424083834.15518-3-pierre-eric.pelloux-prayer@amd.com> References: <20250424083834.15518-1-pierre-eric.pelloux-prayer@amd.com> <20250424083834.15518-3-pierre-eric.pelloux-prayer@amd.com> Message-ID: On Thu, 2025-04-24 at 10:38 +0200, Pierre-Eric Pelloux-Prayer wrote: > This will be used in a later commit to trace the drm client_id in > some of the gpu_scheduler trace events. > > This requires changing all the users of drm_sched_job_init to > add an extra parameter. > > The newly added drm_client_id field in the drm_sched_fence is a bit > of a duplicate of the owner one. One suggestion I received was to > merge those 2 fields - this can't be done right now as amdgpu uses > some special values (AMDGPU_FENCE_OWNER_*) that can't really be > translated into a client id. Christian is working on getting rid of > those; when it's done we should be able to squash owner/drm_client_id > together. > > Reviewed-by: Christian K?nig > Signed-off-by: Pierre-Eric Pelloux-Prayer > > --- > ?drivers/accel/amdxdna/aie2_ctx.c???????????????? |? 3 ++- > ?drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c?????? |? 2 +- > ?drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c?????????? |? 3 ++- > ?drivers/gpu/drm/amd/amdgpu/amdgpu_job.c????????? |? 8 +++++--- > ?drivers/gpu/drm/amd/amdgpu/amdgpu_job.h????????? |? 3 ++- > ?drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c???? |? 2 +- > ?drivers/gpu/drm/imagination/pvr_job.c??????????? |? 2 +- > ?drivers/gpu/drm/imagination/pvr_queue.c????????? |? 5 +++-- > ?drivers/gpu/drm/imagination/paac???????????? |? 8 +++++--- > ?drivers/gpu/drm/nouveau/nouveau_sched.c????????? |? 3 ++- > ?drivers/gpu/drm/panfrost/panfrost_drv.c????????? |? 2 +- > ?drivers/gpu/drm/panthor/panthor_drv.c??????????? |? 3 ++- > ?drivers/gpu/drm/panthor/panthor_mmu.c??????????? |? 2 +- > ?drivers/gpu/drm/panthor/panthor_sched.c????????? |? 5 +++-- > ?drivers/gpu/drm/panthor/panthor_sched.h????????? |? 3 ++- > ?drivers/gpu/drm/scheduler/sched_fence.c????????? |? 4 ++ > ?drivers/gpu/drm/scheduler/sched_internal.h?????? |? 2 +- > ?drivers/gpu/drm/scheduler/sched_main.c?????????? |? 6 ++++-- > ?drivers/gpu/drm/scheduler/tests/mock_scheduler.c |? 2 +- > ?drivers/gpu/drm/v3d/v3d_submit.c???????????????? |? 2 +- > ?drivers/gpu/drm/xe/xe_sched_job.c??????????????? |? 3 ++- > ?include/drm/gpu_scheduler.h????????????????????? | 10 +++++++++- > ?26 files changed, 62 insertions(+), 34 deletions(-) I think last time I asked about what your merge plan for this is, since it touches so many drivers. Should I take that? Besides one comment below, scheduler bits look fine. > > diff --git a/drivers/accel/amdxdna/aie2_ctx.c > b/drivers/accel/amdxdna/aie2_ctx.c > index e04549f64d69..3e38a5f637ea 100644 > --- a/drivers/accel/amdxdna/aie2_ctx.c > +++ b/drivers/accel/amdxdna/aie2_ctx.c > @@ -848,7 +848,8 @@ int aie2_cmd_submit(struct amdxdna_hwctx *hwctx, > struct amdxdna_sched_job *job, > ? goto up_sem; > ? } > ? > - ret = drm_sched_job_init(&job->base, &hwctx->priv->entity, > 1, hwctx); > + ret = drm_sched_job_init(&job->base, &hwctx->priv->entity, > 1, hwctx, > + hwctx->client->filp->client_id); > ? if (ret) { > ? XDNA_ERR(xdna, "DRM job init failed, ret %d", ret); > ? goto free_chain; > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c > b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c > index 4cec3a873995..1a77ba7036c9 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c > @@ -639,7 +639,7 @@ int amdgpu_amdkfd_submit_ib(struct amdgpu_device > *adev, > ? goto err; > ? } > ? > - ret = amdgpu_job_alloc(adev, NULL, NULL, NULL, 1, &job); > + ret = amdgpu_job_alloc(adev, NULL, NULL, NULL, 1, &job, 0); > ? if (ret) > ? goto err; > ? > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c > b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c > index 82df06a72ee0..5a231b997d65 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c > @@ -293,7 +293,8 @@ static int amdgpu_cs_pass1(struct > amdgpu_cs_parser *p, > ? > ? for (i = 0; i < p->gang_size; ++i) { > ? ret = amdgpu_job_alloc(p->adev, vm, p->entities[i], > vm, > - ?????? num_ibs[i], &p->jobs[i]); > + ?????? num_ibs[i], &p->jobs[i], > + ?????? p->filp->client_id); > ? if (ret) > ? goto free_all_kdata; > ? p->jobs[i]->enforce_isolation = p->adev- > >enforce_isolation[fpriv->xcp_id]; > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c > b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c > index acb21fc8b3ce..75262ce8db27 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c > @@ -204,7 +204,8 @@ static enum drm_gpu_sched_stat > amdgpu_job_timedout(struct drm_sched_job *s_job) > ? > ?int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm > *vm, > ? ???? struct drm_sched_entity *entity, void *owner, > - ???? unsigned int num_ibs, struct amdgpu_job **job) > + ???? unsigned int num_ibs, struct amdgpu_job **job, > + ???? u64 drm_client_id) > ?{ > ? if (num_ibs == 0) > ? return -EINVAL; > @@ -222,7 +223,8 @@ int amdgpu_job_alloc(struct amdgpu_device *adev, > struct amdgpu_vm *vm, > ? if (!entity) > ? return 0; > ? > - return drm_sched_job_init(&(*job)->base, entity, 1, owner); > + return drm_sched_job_init(&(*job)->base, entity, 1, owner, > + ? drm_client_id); > ?} > ? > ?int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, > @@ -232,7 +234,7 @@ int amdgpu_job_alloc_with_ib(struct amdgpu_device > *adev, > ?{ > ? int r; > ? > - r = amdgpu_job_alloc(adev, NULL, entity, owner, 1, job); > + r = amdgpu_job_alloc(adev, NULL, entity, owner, 1, job, 0); > ? if (r) > ? return r; > ? > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h > b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h > index ce6b9ba967ff..5a8bc6342222 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h > @@ -90,7 +90,8 @@ static inline struct amdgpu_ring > *amdgpu_job_ring(struct amdgpu_job *job) > ? > ?int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm > *vm, > ? ???? struct drm_sched_entity *entity, void *owner, > - ???? unsigned int num_ibs, struct amdgpu_job **job); > + ???? unsigned int num_ibs, struct amdgpu_job **job, > + ???? u64 drm_client_id); > ?int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, > ? ???? struct drm_sched_entity *entity, void > *owner, > ? ???? size_t size, enum amdgpu_ib_pool_type > pool_type, > diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c > b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c > index 3c0a5c3e0e3d..76c742328edb 100644 > --- a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c > +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c > @@ -534,7 +534,7 @@ int etnaviv_ioctl_gem_submit(struct drm_device > *dev, void *data, > ? > ? ret = drm_sched_job_init(&submit->sched_job, > ? &ctx->sched_entity[args->pipe], > - 1, submit->ctx); > + 1, submit->ctx, file->client_id); > ? if (ret) > ? goto err_submit_put; > ? > diff --git a/drivers/gpu/drm/imagination/pvr_job.c > b/drivers/gpu/drm/imagination/pvr_job.c > index 59b334d094fa..7564b0f21b42 100644 > --- a/drivers/gpu/drm/imagination/pvr_job.c > +++ b/drivers/gpu/drm/imagination/pvr_job.c > @@ -446,7 +446,7 @@ create_job(struct pvr_device *pvr_dev, > ? if (err) > ? goto err_put_job; > ? > - err = pvr_queue_job_init(job); > + err = pvr_queue_job_init(job, pvr_file->file->client_id); > ? if (err) > ? goto err_put_job; > ? > diff --git a/drivers/gpu/drm/imagination/pvr_queue.c > b/drivers/gpu/drm/imagination/pvr_queue.c > index 5e9bc0992824..5a41ee79fed6 100644 > --- a/drivers/gpu/drm/imagination/pvr_queue.c > +++ b/drivers/gpu/drm/imagination/pvr_queue.c > @@ -1073,6 +1073,7 @@ static int pvr_queue_cleanup_fw_context(struct > pvr_queue *queue) > ?/** > ? * pvr_queue_job_init() - Initialize queue related fields in a > pvr_job object. > ? * @job: The job to initialize. > + * @drm_client_id: drm_file.client_id submitting the job > ? * > ? * Bind the job to a queue and allocate memory to guarantee > pvr_queue_job_arm() > ? * and pvr_queue_job_push() can't fail. We also make sure the > context type is > @@ -1082,7 +1083,7 @@ static int pvr_queue_cleanup_fw_context(struct > pvr_queue *queue) > ? *? * 0 on success, or > ? *? * An error code if something failed. > ? */ > -int pvr_queue_job_init(struct pvr_job *job) > +int pvr_queue_job_init(struct pvr_job *job, u64 drm_client_id) > ?{ > ? /* Fragment jobs need at least one native fence wait on the > geometry job fence. */ > ? u32 min_native_dep_count = job->type == > DRM_PVR_JOB_TYPE_FRAGMENT ? 1 : 0; > @@ -1099,7 +1100,7 @@ int pvr_queue_job_init(struct pvr_job *job) > ? if (!pvr_cccb_cmdseq_can_fit(&queue->cccb, > job_cmds_size(job, min_native_dep_count))) > ? return -E2BIG; > ? > - err = drm_sched_job_init(&job->base, &queue->entity, 1, > THIS_MODULE); > + err = drm_sched_job_init(&job->base, &queue->entity, 1, > THIS_MODULE, drm_client_id); > ? if (err) > ? return err; > ? > diff --git a/drivers/gpu/drm/imagination/pvr_queue.h > b/drivers/gpu/drm/imagination/pvr_queue.h > index 93fe9ac9f58c..fc1986d73fc8 100644 > --- a/drivers/gpu/drm/imagination/pvr_queue.h > +++ b/drivers/gpu/drm/imagination/pvr_queue.h > @@ -143,7 +143,7 @@ struct pvr_queue { > ? > ?bool pvr_queue_fence_is_ufo_backed(struct dma_fence *f); > ? > -int pvr_queue_job_init(struct pvr_job *job); > +int pvr_queue_job_init(struct pvr_job *job, u64 drm_client_id); > ? > ?void pvr_queue_job_cleanup(struct pvr_job *job); > ? > diff --git a/drivers/gpu/drm/lima/lima_gem.c > b/drivers/gpu/drm/lima/lima_gem.c > index 5deec673c11e..9722b847a539 100644 > --- a/drivers/gpu/drm/lima/lima_gem.c > +++ b/drivers/gpu/drm/lima/lima_gem.c > @@ -341,7 +341,7 @@ int lima_gem_submit(struct drm_file *file, struct > lima_submit *submit) > ? > ? err = lima_sched_task_init( > ? submit->task, submit->ctx->context + submit->pipe, > - bos, submit->nr_bos, vm); > + bos, submit->nr_bos, vm, file->client_id); > ? if (err) > ? goto err_out1; > ? > diff --git a/drivers/gpu/drm/lima/lima_sched.c > b/drivers/gpu/drm/lima/lima_sched.c > index 7934098e651b..954f4325b859 100644 > --- a/drivers/gpu/drm/lima/lima_sched.c > +++ b/drivers/gpu/drm/lima/lima_sched.c > @@ -113,7 +113,8 @@ static inline struct lima_sched_pipe > *to_lima_pipe(struct drm_gpu_scheduler *sch > ?int lima_sched_task_init(struct lima_sched_task *task, > ? struct lima_sched_context *context, > ? struct lima_bo **bos, int num_bos, > - struct lima_vm *vm) > + struct lima_vm *vm, > + u64 drm_client_id) > ?{ > ? int err, i; > ? > @@ -124,7 +125,8 @@ int lima_sched_task_init(struct lima_sched_task > *task, > ? for (i = 0; i < num_bos; i++) > ? drm_gem_object_get(&bos[i]->base.base); > ? > - err = drm_sched_job_init(&task->base, &context->base, 1, > vm); > + err = drm_sched_job_init(&task->base, &context->base, 1, vm, > + drm_client_id); > ? if (err) { > ? kfree(task->bos); > ? return err; > diff --git a/drivers/gpu/drm/lima/lima_sched.h > b/drivers/gpu/drm/lima/lima_sched.h > index 85b23ba901d5..1a08faf8a529 100644 > --- a/drivers/gpu/drm/lima/lima_sched.h > +++ b/drivers/gpu/drm/lima/lima_sched.h > @@ -88,7 +88,8 @@ struct lima_sched_pipe { > ?int lima_sched_task_init(struct lima_sched_task *task, > ? struct lima_sched_context *context, > ? struct lima_bo **bos, int num_bos, > - struct lima_vm *vm); > + struct lima_vm *vm, > + u64 drm_client_id); > ?void lima_sched_task_fini(struct lima_sched_task *task); > ? > ?int lima_sched_context_init(struct lima_sched_pipe *pipe, > diff --git a/drivers/gpu/drm/msm/msm_gem_submit.c > b/drivers/gpu/drm/msm/msm_gem_submit.c > index 3e9aa2cc38ef..d9be0fe3d674 100644 > --- a/drivers/gpu/drm/msm/msm_gem_submit.c > +++ b/drivers/gpu/drm/msm/msm_gem_submit.c > @@ -30,7 +30,7 @@ > ?static struct msm_gem_submit *submit_create(struct drm_device *dev, > ? struct msm_gpu *gpu, > ? struct msm_gpu_submitqueue *queue, uint32_t nr_bos, > - uint32_t nr_cmds) > + uint32_t nr_cmds, u64 drm_client_id) > ?{ > ? static atomic_t ident = ATOMIC_INIT(0); > ? struct msm_gem_submit *submit; > @@ -54,7 +54,8 @@ static struct msm_gem_submit *submit_create(struct > drm_device *dev, > ? return ERR_PTR(ret); > ? } > ? > - ret = drm_sched_job_init(&submit->base, queue->entity, 1, > queue); > + ret = drm_sched_job_init(&submit->base, queue->entity, 1, > queue, > + drm_client_id); > ? if (ret) { > ? kfree(submit->hw_fence); > ? kfree(submit); > @@ -693,7 +694,8 @@ int msm_ioctl_gem_submit(struct drm_device *dev, > void *data, > ? } > ? } > ? > - submit = submit_create(dev, gpu, queue, args->nr_bos, args- > >nr_cmds); > + submit = submit_create(dev, gpu, queue, args->nr_bos, args- > >nr_cmds, > + ?????? file->client_id); > ? if (IS_ERR(submit)) { > ? ret = PTR_ERR(submit); > ? goto out_post_unlock; > diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.c > b/drivers/gpu/drm/nouveau/nouveau_sched.c > index d326e55d2d24..460a5fb02412 100644 > --- a/drivers/gpu/drm/nouveau/nouveau_sched.c > +++ b/drivers/gpu/drm/nouveau/nouveau_sched.c > @@ -87,7 +87,8 @@ nouveau_job_init(struct nouveau_job *job, > ? } > ? > ? ret = drm_sched_job_init(&job->base, &sched->entity, > - args->credits, NULL); > + args->credits, NULL, > + job->file_priv->client_id); > ? if (ret) > ? goto err_free_chains; > ? > diff --git a/drivers/gpu/drm/panfrost/panfrost_drv.c > b/drivers/gpu/drm/panfrost/panfrost_drv.c > index b87f83e94eda..d5c2c6530ed8 100644 > --- a/drivers/gpu/drm/panfrost/panfrost_drv.c > +++ b/drivers/gpu/drm/panfrost/panfrost_drv.c > @@ -312,7 +312,7 @@ static int panfrost_ioctl_submit(struct > drm_device *dev, void *data, > ? > ? ret = drm_sched_job_init(&job->base, > ? &file_priv->sched_entity[slot], > - 1, NULL); > + 1, NULL, file->client_id); > ? if (ret) > ? goto out_put_job; > ? > diff --git a/drivers/gpu/drm/panthor/panthor_drv.c > b/drivers/gpu/drm/panthor/panthor_drv.c > index 06fe46e32073..bd8e1900c919 100644 > --- a/drivers/gpu/drm/panthor/panthor_drv.c > +++ b/drivers/gpu/drm/panthor/panthor_drv.c > @@ -989,7 +989,8 @@ static int panthor_ioctl_group_submit(struct > drm_device *ddev, void *data, > ? const struct drm_panthor_queue_submit *qsubmit = > &jobs_args[i]; > ? struct drm_sched_job *job; > ? > - job = panthor_job_create(pfile, args->group_handle, > qsubmit); > + job = panthor_job_create(pfile, args->group_handle, > qsubmit, > + file->client_id); > ? if (IS_ERR(job)) { > ? ret = PTR_ERR(job); > ? goto out_cleanup_submit_ctx; > diff --git a/drivers/gpu/drm/panthor/panthor_mmu.c > b/drivers/gpu/drm/panthor/panthor_mmu.c > index 12a02e28f50f..e0c79bd2d173 100644 > --- a/drivers/gpu/drm/panthor/panthor_mmu.c > +++ b/drivers/gpu/drm/panthor/panthor_mmu.c > @@ -2516,7 +2516,7 @@ panthor_vm_bind_job_create(struct drm_file > *file, > ? kref_init(&job->refcount); > ? job->vm = panthor_vm_get(vm); > ? > - ret = drm_sched_job_init(&job->base, &vm->entity, 1, vm); > + ret = drm_sched_job_init(&job->base, &vm->entity, 1, vm, > file->client_id); > ? if (ret) > ? goto err_put_job; > ? > diff --git a/drivers/gpu/drm/panthor/panthor_sched.c > b/drivers/gpu/drm/panthor/panthor_sched.c > index 446ec780eb4a..2af860c9068a 100644 > --- a/drivers/gpu/drm/panthor/panthor_sched.c > +++ b/drivers/gpu/drm/panthor/panthor_sched.c > @@ -3729,7 +3729,8 @@ struct panthor_vm *panthor_job_vm(struct > drm_sched_job *sched_job) > ?struct drm_sched_job * > ?panthor_job_create(struct panthor_file *pfile, > ? ?? u16 group_handle, > - ?? const struct drm_panthor_queue_submit *qsubmit) > + ?? const struct drm_panthor_queue_submit *qsubmit, > + ?? u64 drm_client_id) > ?{ > ? struct panthor_group_pool *gpool = pfile->groups; > ? struct panthor_job *job; > @@ -3801,7 +3802,7 @@ panthor_job_create(struct panthor_file *pfile, > ? > ? ret = drm_sched_job_init(&job->base, > ? &job->group->queues[job- > >queue_idx]->entity, > - credits, job->group); > + credits, job->group, > drm_client_id); > ? if (ret) > ? goto err_put_job; > ? > diff --git a/drivers/gpu/drm/panthor/panthor_sched.h > b/drivers/gpu/drm/panthor/panthor_sched.h > index e650a445cf50..742b0b4ff3a3 100644 > --- a/drivers/gpu/drm/panthor/panthor_sched.h > +++ b/drivers/gpu/drm/panthor/panthor_sched.h > @@ -29,7 +29,8 @@ int panthor_group_get_state(struct panthor_file > *pfile, > ?struct drm_sched_job * > ?panthor_job_create(struct panthor_file *pfile, > ? ?? u16 group_handle, > - ?? const struct drm_panthor_queue_submit *qsubmit); > + ?? const struct drm_panthor_queue_submit *qsubmit, > + ?? u64 drm_client_id); > ?struct drm_sched_job *panthor_job_get(struct drm_sched_job *job); > ?struct panthor_vm *panthor_job_vm(struct drm_sched_job *sched_job); > ?void panthor_job_put(struct drm_sched_job *job); > diff --git a/drivers/gpu/drm/scheduler/sched_fence.c > b/drivers/gpu/drm/scheduler/sched_fence.c > index e971528504a5..d208d384d38d 100644 > --- a/drivers/gpu/drm/scheduler/sched_fence.c > +++ b/drivers/gpu/drm/scheduler/sched_fence.c > @@ -206,7 +206,8 @@ struct drm_sched_fence *to_drm_sched_fence(struct > dma_fence *f) > ?EXPORT_SYMBOL(to_drm_sched_fence); > ? > ?struct drm_sched_fence *drm_sched_fence_alloc(struct > drm_sched_entity *entity, > - ????? void *owner) > + ????? void *owner, > + ????? u64 drm_client_id) > ?{ > ? struct drm_sched_fence *fence = NULL; > ? > @@ -215,6 +216,7 @@ struct drm_sched_fence > *drm_sched_fence_alloc(struct drm_sched_entity *entity, > ? return NULL; > ? > ? fence->owner = owner; > + fence->drm_client_id = drm_client_id; > ? spin_lock_init(&fence->lock); > ? > ? return fence; > diff --git a/drivers/gpu/drm/scheduler/sched_internal.h > b/drivers/gpu/drm/scheduler/sched_internal.h > index 599cf6e1bb74..7ea5a6736f98 100644 > --- a/drivers/gpu/drm/scheduler/sched_internal.h > +++ b/drivers/gpu/drm/scheduler/sched_internal.h > @@ -24,7 +24,7 @@ void drm_sched_entity_select_rq(struct > drm_sched_entity *entity); > ?struct drm_sched_job *drm_sched_entity_pop_job(struct > drm_sched_entity *entity); > ? > ?struct drm_sched_fence *drm_sched_fence_alloc(struct > drm_sched_entity *s_entity, > - ????? void *owner); > + ????? void *owner, u64 > drm_client_id); > ?void drm_sched_fence_init(struct drm_sched_fence *fence, > ? ? struct drm_sched_entity *entity); > ?void drm_sched_fence_free(struct drm_sched_fence *fence); > diff --git a/drivers/gpu/drm/scheduler/sched_main.c > b/drivers/gpu/drm/scheduler/sched_main.c > index 829579c41c6b..60611618f3ab 100644 > --- a/drivers/gpu/drm/scheduler/sched_main.c > +++ b/drivers/gpu/drm/scheduler/sched_main.c > @@ -764,6 +764,7 @@ EXPORT_SYMBOL(drm_sched_resubmit_jobs); > ? * @credits: the number of credits this job contributes to the > schedulers > ? * credit limit > ? * @owner: job owner for debugging > + * @drm_client_id: drm_file.client_id of the owner For the docu generation to link that properly it must be written as &struct drm_file.client_id Besides, if this were an optional parameter, one should document it. I'm not sure if it is, I haven't used these client_id's before. P. > ? * > ? * Refer to drm_sched_entity_push_job() documentation > ? * for locking considerations. > @@ -784,7 +785,8 @@ EXPORT_SYMBOL(drm_sched_resubmit_jobs); > ? */ > ?int drm_sched_job_init(struct drm_sched_job *job, > ? ?????? struct drm_sched_entity *entity, > - ?????? u32 credits, void *owner) > + ?????? u32 credits, void *owner, > + ?????? uint64_t drm_client_id) > ?{ > ? if (!entity->rq) { > ? /* This will most likely be followed by missing > frames > @@ -810,7 +812,7 @@ int drm_sched_job_init(struct drm_sched_job *job, > ? > ? job->entity = entity; > ? job->credits = credits; > - job->s_fence = drm_sched_fence_alloc(entity, owner); > + job->s_fence = drm_sched_fence_alloc(entity, owner, > drm_client_id); > ? if (!job->s_fence) > ? return -ENOMEM; > ? > diff --git a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c > b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c > index f999c8859cf7..09ffbdb32d76 100644 > --- a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c > +++ b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c > @@ -35,7 +35,7 @@ drm_mock_sched_entity_new(struct kunit *test, > ? ret = drm_sched_entity_init(&entity->base, > ? ??? priority, > ? ??? &drm_sched, 1, > - ??? NULL); > + ??? NULL, 1); > ? KUNIT_ASSERT_EQ(test, ret, 0); > ? > ? entity->test = test; > diff --git a/drivers/gpu/drm/v3d/v3d_submit.c > b/drivers/gpu/drm/v3d/v3d_submit.c > index 4ff5de46fb22..5171ffe9012d 100644 > --- a/drivers/gpu/drm/v3d/v3d_submit.c > +++ b/drivers/gpu/drm/v3d/v3d_submit.c > @@ -169,7 +169,7 @@ v3d_job_init(struct v3d_dev *v3d, struct drm_file > *file_priv, > ? job->file = file_priv; > ? > ? ret = drm_sched_job_init(&job->base, &v3d_priv- > >sched_entity[queue], > - 1, v3d_priv); > + 1, v3d_priv, file_priv->client_id); > ? if (ret) > ? return ret; > ? > diff --git a/drivers/gpu/drm/xe/xe_sched_job.c > b/drivers/gpu/drm/xe/xe_sched_job.c > index 1905ca590965..f4679cb9a56b 100644 > --- a/drivers/gpu/drm/xe/xe_sched_job.c > +++ b/drivers/gpu/drm/xe/xe_sched_job.c > @@ -113,7 +113,8 @@ struct xe_sched_job *xe_sched_job_create(struct > xe_exec_queue *q, > ? kref_init(&job->refcount); > ? xe_exec_queue_get(job->q); > ? > - err = drm_sched_job_init(&job->drm, q->entity, 1, NULL); > + err = drm_sched_job_init(&job->drm, q->entity, 1, NULL, > + q->xef->drm->client_id); > ? if (err) > ? goto err_free; > ? > diff --git a/include/drm/gpu_scheduler.h > b/include/drm/gpu_scheduler.h > index 1a7e377d4cbb..6fe3b4c0cffb 100644 > --- a/include/drm/gpu_scheduler.h > +++ b/include/drm/gpu_scheduler.h > @@ -305,6 +305,13 @@ struct drm_sched_fence { > ????????? * @owner: job owner for debugging > ????????? */ > ? void *owner; > + > + /** > + * @drm_client_id: > + * > + * The client_id of the drm_file which owns the job. > + */ > + uint64_t drm_client_id; > ?}; > ? > ?struct drm_sched_fence *to_drm_sched_fence(struct dma_fence *f); > @@ -629,7 +636,8 @@ drm_sched_pick_best(struct drm_gpu_scheduler > **sched_list, > ? > ?int drm_sched_job_init(struct drm_sched_job *job, > ? ?????? struct drm_sched_entity *entity, > - ?????? u32 credits, void *owner); > + ?????? u32 credits, void *owner, > + ?????? u64 drm_client_id); > ?void drm_sched_job_arm(struct drm_sched_job *job); > ?void drm_sched_entity_push_job(struct drm_sched_job *sched_job); > ?int drm_sched_job_add_dependency(struct drm_sched_job *job, From pierre-eric at damsy.net Thu May 15 06:53:28 2025 From: pierre-eric at damsy.net (Pierre-Eric Pelloux-Prayer) Date: Thu, 15 May 2025 08:53:28 +0200 Subject: [PATCH v9 02/10] drm/sched: store the drm client_id in drm_sched_fence In-Reply-To: References: <20250424083834.15518-1-pierre-eric.pelloux-prayer@amd.com> <20250424083834.15518-3-pierre-eric.pelloux-prayer@amd.com> Message-ID: <2d206814-968d-47df-b670-cbd8db68c5c6@damsy.net> Hi, Le 14/05/2025 ? 14:44, Philipp Stanner a ?crit?: > On Thu, 2025-04-24 at 10:38 +0200, Pierre-Eric Pelloux-Prayer wrote: >> This will be used in a later commit to trace the drm client_id in >> some of the gpu_scheduler trace events. >> >> This requires changing all the users of drm_sched_job_init to >> add an extra parameter. >> >> The newly added drm_client_id field in the drm_sched_fence is a bit >> of a duplicate of the owner one. One suggestion I received was to >> merge those 2 fields - this can't be done right now as amdgpu uses >> some special values (AMDGPU_FENCE_OWNER_*) that can't really be >> translated into a client id. Christian is working on getting rid of >> those; when it's done we should be able to squash owner/drm_client_id >> together. >> >> Reviewed-by: Christian K?nig >> Signed-off-by: Pierre-Eric Pelloux-Prayer >> >> --- >> ?drivers/accel/amdxdna/aie2_ctx.c???????????????? |? 3 ++- >> ?drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c?????? |? 2 +- >> ?drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c?????????? |? 3 ++- >> ?drivers/gpu/drm/amd/amdgpu/amdgpu_job.c????????? |? 8 +++++--- >> ?drivers/gpu/drm/amd/amdgpu/amdgpu_job.h????????? |? 3 ++- >> ?drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c???? |? 2 +- >> ?drivers/gpu/drm/imagination/pvr_job.c??????????? |? 2 +- >> ?drivers/gpu/drm/imagination/pvr_queue.c????????? |? 5 +++-- >> ?drivers/gpu/drm/imagination/paac???????????? |? 8 +++++--- >> ?drivers/gpu/drm/nouveau/nouveau_sched.c????????? |? 3 ++- >> ?drivers/gpu/drm/panfrost/panfrost_drv.c????????? |? 2 +- >> ?drivers/gpu/drm/panthor/panthor_drv.c??????????? |? 3 ++- >> ?drivers/gpu/drm/panthor/panthor_mmu.c??????????? |? 2 +- >> ?drivers/gpu/drm/panthor/panthor_sched.c????????? |? 5 +++-- >> ?drivers/gpu/drm/panthor/panthor_sched.h????????? |? 3 ++- >> ?drivers/gpu/drm/scheduler/sched_fence.c????????? |? 4 ++ >> ?drivers/gpu/drm/scheduler/sched_internal.h?????? |? 2 +- >> ?drivers/gpu/drm/scheduler/sched_main.c?????????? |? 6 ++++-- >> ?drivers/gpu/drm/scheduler/tests/mock_scheduler.c |? 2 +- >> ?drivers/gpu/drm/v3d/v3d_submit.c???????????????? |? 2 +- >> ?drivers/gpu/drm/xe/xe_sched_job.c??????????????? |? 3 ++- >> ?include/drm/gpu_scheduler.h????????????????????? | 10 +++++++++- >> ?26 files changed, 62 insertions(+), 34 deletions(-) > > I think last time I asked about what your merge plan for this is, since > it touches so many drivers. Should I take that? Based on: https://drm.pages.freedesktop.org/maintainer-tools/committer/committer-drm-misc.html "drm-misc is for drm core (non-driver) patches, subsystem-wide refactorings, and small trivial patches all over (including drivers)." I assume it should go through drm-misc. > > Besides one comment below, scheduler bits look fine. > >> >> diff --git a/drivers/accel/amdxdna/aie2_ctx.c >> b/drivers/accel/amdxdna/aie2_ctx.c >> index e04549f64d69..3e38a5f637ea 100644 >> --- a/drivers/accel/amdxdna/aie2_ctx.c >> +++ b/drivers/accel/amdxdna/aie2_ctx.c >> @@ -848,7 +848,8 @@ int aie2_cmd_submit(struct amdxdna_hwctx *hwctx, >> struct amdxdna_sched_job *job, >> ? goto up_sem; >> ? } >> >> - ret = drm_sched_job_init(&job->base, &hwctx->priv->entity, >> 1, hwctx); >> + ret = drm_sched_job_init(&job->base, &hwctx->priv->entity, >> 1, hwctx, >> + hwctx->client->filp->client_id); >> ? if (ret) { >> ? XDNA_ERR(xdna, "DRM job init failed, ret %d", ret); >> ? goto free_chain; >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c >> index 4cec3a873995..1a77ba7036c9 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c >> @@ -639,7 +639,7 @@ int amdgpu_amdkfd_submit_ib(struct amdgpu_device >> *adev, >> ? goto err; >> ? } >> >> - ret = amdgpu_job_alloc(adev, NULL, NULL, NULL, 1, &job); >> + ret = amdgpu_job_alloc(adev, NULL, NULL, NULL, 1, &job, 0); >> ? if (ret) >> ? goto err; >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c >> index 82df06a72ee0..5a231b997d65 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c >> @@ -293,7 +293,8 @@ static int amdgpu_cs_pass1(struct >> amdgpu_cs_parser *p, >> >> ? for (i = 0; i < p->gang_size; ++i) { >> ? ret = amdgpu_job_alloc(p->adev, vm, p->entities[i], >> vm, >> - ?????? num_ibs[i], &p->jobs[i]); >> + ?????? num_ibs[i], &p->jobs[i], >> + ?????? p->filp->client_id); >> ? if (ret) >> ? goto free_all_kdata; >> ? p->jobs[i]->enforce_isolation = p->adev- >>> enforce_isolation[fpriv->xcp_id]; >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c >> index acb21fc8b3ce..75262ce8db27 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c >> @@ -204,7 +204,8 @@ static enum drm_gpu_sched_stat >> amdgpu_job_timedout(struct drm_sched_job *s_job) >> >> ?int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm >> *vm, >> ? ???? struct drm_sched_entity *entity, void *owner, >> - ???? unsigned int num_ibs, struct amdgpu_job **job) >> + ???? unsigned int num_ibs, struct amdgpu_job **job, >> + ???? u64 drm_client_id) >> ?{ >> ? if (num_ibs == 0) >> ? return -EINVAL; >> @@ -222,7 +223,8 @@ int amdgpu_job_alloc(struct amdgpu_device *adev, >> struct amdgpu_vm *vm, >> ? if (!entity) >> ? return 0; >> >> - return drm_sched_job_init(&(*job)->base, entity, 1, owner); >> + return drm_sched_job_init(&(*job)->base, entity, 1, owner, >> + ? drm_client_id); >> ?} >> >> ?int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, >> @@ -232,7 +234,7 @@ int amdgpu_job_alloc_with_ib(struct amdgpu_device >> *adev, >> ?{ >> ? int r; >> >> - r = amdgpu_job_alloc(adev, NULL, entity, owner, 1, job); >> + r = amdgpu_job_alloc(adev, NULL, entity, owner, 1, job, 0); >> ? if (r) >> ? return r; >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h >> index ce6b9ba967ff..5a8bc6342222 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h >> @@ -90,7 +90,8 @@ static inline struct amdgpu_ring >> *amdgpu_job_ring(struct amdgpu_job *job) >> >> ?int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm >> *vm, >> ? ???? struct drm_sched_entity *entity, void *owner, >> - ???? unsigned int num_ibs, struct amdgpu_job **job); >> + ???? unsigned int num_ibs, struct amdgpu_job **job, >> + ???? u64 drm_client_id); >> ?int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, >> ? ???? struct drm_sched_entity *entity, void >> *owner, >> ? ???? size_t size, enum amdgpu_ib_pool_type >> pool_type, >> diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c >> b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c >> index 3c0a5c3e0e3d..76c742328edb 100644 >> --- a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c >> +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c >> @@ -534,7 +534,7 @@ int etnaviv_ioctl_gem_submit(struct drm_device >> *dev, void *data, >> >> ? ret = drm_sched_job_init(&submit->sched_job, >> ? &ctx->sched_entity[args->pipe], >> - 1, submit->ctx); >> + 1, submit->ctx, file->client_id); >> ? if (ret) >> ? goto err_submit_put; >> >> diff --git a/drivers/gpu/drm/imagination/pvr_job.c >> b/drivers/gpu/drm/imagination/pvr_job.c >> index 59b334d094fa..7564b0f21b42 100644 >> --- a/drivers/gpu/drm/imagination/pvr_job.c >> +++ b/drivers/gpu/drm/imagination/pvr_job.c >> @@ -446,7 +446,7 @@ create_job(struct pvr_device *pvr_dev, >> ? if (err) >> ? goto err_put_job; >> >> - err = pvr_queue_job_init(job); >> + err = pvr_queue_job_init(job, pvr_file->file->client_id); >> ? if (err) >> ? goto err_put_job; >> >> diff --git a/drivers/gpu/drm/imagination/pvr_queue.c >> b/drivers/gpu/drm/imagination/pvr_queue.c >> index 5e9bc0992824..5a41ee79fed6 100644 >> --- a/drivers/gpu/drm/imagination/pvr_queue.c >> +++ b/drivers/gpu/drm/imagination/pvr_queue.c >> @@ -1073,6 +1073,7 @@ static int pvr_queue_cleanup_fw_context(struct >> pvr_queue *queue) >> ?/** >> ? * pvr_queue_job_init() - Initialize queue related fields in a >> pvr_job object. >> ? * @job: The job to initialize. >> + * @drm_client_id: drm_file.client_id submitting the job >> ? * >> ? * Bind the job to a queue and allocate memory to guarantee >> pvr_queue_job_arm() >> ? * and pvr_queue_job_push() can't fail. We also make sure the >> context type is >> @@ -1082,7 +1083,7 @@ static int pvr_queue_cleanup_fw_context(struct >> pvr_queue *queue) >> ? *? * 0 on success, or >> ? *? * An error code if something failed. >> ? */ >> -int pvr_queue_job_init(struct pvr_job *job) >> +int pvr_queue_job_init(struct pvr_job *job, u64 drm_client_id) >> ?{ >> ? /* Fragment jobs need at least one native fence wait on the >> geometry job fence. */ >> ? u32 min_native_dep_count = job->type == >> DRM_PVR_JOB_TYPE_FRAGMENT ? 1 : 0; >> @@ -1099,7 +1100,7 @@ int pvr_queue_job_init(struct pvr_job *job) >> ? if (!pvr_cccb_cmdseq_can_fit(&queue->cccb, >> job_cmds_size(job, min_native_dep_count))) >> ? return -E2BIG; >> >> - err = drm_sched_job_init(&job->base, &queue->entity, 1, >> THIS_MODULE); >> + err = drm_sched_job_init(&job->base, &queue->entity, 1, >> THIS_MODULE, drm_client_id); >> ? if (err) >> ? return err; >> >> diff --git a/drivers/gpu/drm/imagination/pvr_queue.h >> b/drivers/gpu/drm/imagination/pvr_queue.h >> index 93fe9ac9f58c..fc1986d73fc8 100644 >> --- a/drivers/gpu/drm/imagination/pvr_queue.h >> +++ b/drivers/gpu/drm/imagination/pvr_queue.h >> @@ -143,7 +143,7 @@ struct pvr_queue { >> >> ?bool pvr_queue_fence_is_ufo_backed(struct dma_fence *f); >> >> -int pvr_queue_job_init(struct pvr_job *job); >> +int pvr_queue_job_init(struct pvr_job *job, u64 drm_client_id); >> >> ?void pvr_queue_job_cleanup(struct pvr_job *job); >> >> diff --git a/drivers/gpu/drm/lima/lima_gem.c >> b/drivers/gpu/drm/lima/lima_gem.c >> index 5deec673c11e..9722b847a539 100644 >> --- a/drivers/gpu/drm/lima/lima_gem.c >> +++ b/drivers/gpu/drm/lima/lima_gem.c >> @@ -341,7 +341,7 @@ int lima_gem_submit(struct drm_file *file, struct >> lima_submit *submit) >> >> ? err = lima_sched_task_init( >> ? submit->task, submit->ctx->context + submit->pipe, >> - bos, submit->nr_bos, vm); >> + bos, submit->nr_bos, vm, file->client_id); >> ? if (err) >> ? goto err_out1; >> >> diff --git a/drivers/gpu/drm/lima/lima_sched.c >> b/drivers/gpu/drm/lima/lima_sched.c >> index 7934098e651b..954f4325b859 100644 >> --- a/drivers/gpu/drm/lima/lima_sched.c >> +++ b/drivers/gpu/drm/lima/lima_sched.c >> @@ -113,7 +113,8 @@ static inline struct lima_sched_pipe >> *to_lima_pipe(struct drm_gpu_scheduler *sch >> ?int lima_sched_task_init(struct lima_sched_task *task, >> ? struct lima_sched_context *context, >> ? struct lima_bo **bos, int num_bos, >> - struct lima_vm *vm) >> + struct lima_vm *vm, >> + u64 drm_client_id) >> ?{ >> ? int err, i; >> >> @@ -124,7 +125,8 @@ int lima_sched_task_init(struct lima_sched_task >> *task, >> ? for (i = 0; i < num_bos; i++) >> ? drm_gem_object_get(&bos[i]->base.base); >> >> - err = drm_sched_job_init(&task->base, &context->base, 1, >> vm); >> + err = drm_sched_job_init(&task->base, &context->base, 1, vm, >> + drm_client_id); >> ? if (err) { >> ? kfree(task->bos); >> ? return err; >> diff --git a/drivers/gpu/drm/lima/lima_sched.h >> b/drivers/gpu/drm/lima/lima_sched.h >> index 85b23ba901d5..1a08faf8a529 100644 >> --- a/drivers/gpu/drm/lima/lima_sched.h >> +++ b/drivers/gpu/drm/lima/lima_sched.h >> @@ -88,7 +88,8 @@ struct lima_sched_pipe { >> ?int lima_sched_task_init(struct lima_sched_task *task, >> ? struct lima_sched_context *context, >> ? struct lima_bo **bos, int num_bos, >> - struct lima_vm *vm); >> + struct lima_vm *vm, >> + u64 drm_client_id); >> ?void lima_sched_task_fini(struct lima_sched_task *task); >> >> ?int lima_sched_context_init(struct lima_sched_pipe *pipe, >> diff --git a/drivers/gpu/drm/msm/msm_gem_submit.c >> b/drivers/gpu/drm/msm/msm_gem_submit.c >> index 3e9aa2cc38ef..d9be0fe3d674 100644 >> --- a/drivers/gpu/drm/msm/msm_gem_submit.c >> +++ b/drivers/gpu/drm/msm/msm_gem_submit.c >> @@ -30,7 +30,7 @@ >> ?static struct msm_gem_submit *submit_create(struct drm_device *dev, >> ? struct msm_gpu *gpu, >> ? struct msm_gpu_submitqueue *queue, uint32_t nr_bos, >> - uint32_t nr_cmds) >> + uint32_t nr_cmds, u64 drm_client_id) >> ?{ >> ? static atomic_t ident = ATOMIC_INIT(0); >> ? struct msm_gem_submit *submit; >> @@ -54,7 +54,8 @@ static struct msm_gem_submit *submit_create(struct >> drm_device *dev, >> ? return ERR_PTR(ret); >> ? } >> >> - ret = drm_sched_job_init(&submit->base, queue->entity, 1, >> queue); >> + ret = drm_sched_job_init(&submit->base, queue->entity, 1, >> queue, >> + drm_client_id); >> ? if (ret) { >> ? kfree(submit->hw_fence); >> ? kfree(submit); >> @@ -693,7 +694,8 @@ int msm_ioctl_gem_submit(struct drm_device *dev, >> void *data, >> ? } >> ? } >> >> - submit = submit_create(dev, gpu, queue, args->nr_bos, args- >>> nr_cmds); >> + submit = submit_create(dev, gpu, queue, args->nr_bos, args- >>> nr_cmds, >> + ?????? file->client_id); >> ? if (IS_ERR(submit)) { >> ? ret = PTR_ERR(submit); >> ? goto out_post_unlock; >> diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.c >> b/drivers/gpu/drm/nouveau/nouveau_sched.c >> index d326e55d2d24..460a5fb02412 100644 >> --- a/drivers/gpu/drm/nouveau/nouveau_sched.c >> +++ b/drivers/gpu/drm/nouveau/nouveau_sched.c >> @@ -87,7 +87,8 @@ nouveau_job_init(struct nouveau_job *job, >> ? } >> >> ? ret = drm_sched_job_init(&job->base, &sched->entity, >> - args->credits, NULL); >> + args->credits, NULL, >> + job->file_priv->client_id); >> ? if (ret) >> ? goto err_free_chains; >> >> diff --git a/drivers/gpu/drm/panfrost/panfrost_drv.c >> b/drivers/gpu/drm/panfrost/panfrost_drv.c >> index b87f83e94eda..d5c2c6530ed8 100644 >> --- a/drivers/gpu/drm/panfrost/panfrost_drv.c >> +++ b/drivers/gpu/drm/panfrost/panfrost_drv.c >> @@ -312,7 +312,7 @@ static int panfrost_ioctl_submit(struct >> drm_device *dev, void *data, >> >> ? ret = drm_sched_job_init(&job->base, >> ? &file_priv->sched_entity[slot], >> - 1, NULL); >> + 1, NULL, file->client_id); >> ? if (ret) >> ? goto out_put_job; >> >> diff --git a/drivers/gpu/drm/panthor/panthor_drv.c >> b/drivers/gpu/drm/panthor/panthor_drv.c >> index 06fe46e32073..bd8e1900c919 100644 >> --- a/drivers/gpu/drm/panthor/panthor_drv.c >> +++ b/drivers/gpu/drm/panthor/panthor_drv.c >> @@ -989,7 +989,8 @@ static int panthor_ioctl_group_submit(struct >> drm_device *ddev, void *data, >> ? const struct drm_panthor_queue_submit *qsubmit = >> &jobs_args[i]; >> ? struct drm_sched_job *job; >> >> - job = panthor_job_create(pfile, args->group_handle, >> qsubmit); >> + job = panthor_job_create(pfile, args->group_handle, >> qsubmit, >> + file->client_id); >> ? if (IS_ERR(job)) { >> ? ret = PTR_ERR(job); >> ? goto out_cleanup_submit_ctx; >> diff --git a/drivers/gpu/drm/panthor/panthor_mmu.c >> b/drivers/gpu/drm/panthor/panthor_mmu.c >> index 12a02e28f50f..e0c79bd2d173 100644 >> --- a/drivers/gpu/drm/panthor/panthor_mmu.c >> +++ b/drivers/gpu/drm/panthor/panthor_mmu.c >> @@ -2516,7 +2516,7 @@ panthor_vm_bind_job_create(struct drm_file >> *file, >> ? kref_init(&job->refcount); >> ? job->vm = panthor_vm_get(vm); >> >> - ret = drm_sched_job_init(&job->base, &vm->entity, 1, vm); >> + ret = drm_sched_job_init(&job->base, &vm->entity, 1, vm, >> file->client_id); >> ? if (ret) >> ? goto err_put_job; >> >> diff --git a/drivers/gpu/drm/panthor/panthor_sched.c >> b/drivers/gpu/drm/panthor/panthor_sched.c >> index 446ec780eb4a..2af860c9068a 100644 >> --- a/drivers/gpu/drm/panthor/panthor_sched.c >> +++ b/drivers/gpu/drm/panthor/panthor_sched.c >> @@ -3729,7 +3729,8 @@ struct panthor_vm *panthor_job_vm(struct >> drm_sched_job *sched_job) >> ?struct drm_sched_job * >> ?panthor_job_create(struct panthor_file *pfile, >> ? ?? u16 group_handle, >> - ?? const struct drm_panthor_queue_submit *qsubmit) >> + ?? const struct drm_panthor_queue_submit *qsubmit, >> + ?? u64 drm_client_id) >> ?{ >> ? struct panthor_group_pool *gpool = pfile->groups; >> ? struct panthor_job *job; >> @@ -3801,7 +3802,7 @@ panthor_job_create(struct panthor_file *pfile, >> >> ? ret = drm_sched_job_init(&job->base, >> ? &job->group->queues[job- >>> queue_idx]->entity, >> - credits, job->group); >> + credits, job->group, >> drm_client_id); >> ? if (ret) >> ? goto err_put_job; >> >> diff --git a/drivers/gpu/drm/panthor/panthor_sched.h >> b/drivers/gpu/drm/panthor/panthor_sched.h >> index e650a445cf50..742b0b4ff3a3 100644 >> --- a/drivers/gpu/drm/panthor/panthor_sched.h >> +++ b/drivers/gpu/drm/panthor/panthor_sched.h >> @@ -29,7 +29,8 @@ int panthor_group_get_state(struct panthor_file >> *pfile, >> ?struct drm_sched_job * >> ?panthor_job_create(struct panthor_file *pfile, >> ? ?? u16 group_handle, >> - ?? const struct drm_panthor_queue_submit *qsubmit); >> + ?? const struct drm_panthor_queue_submit *qsubmit, >> + ?? u64 drm_client_id); >> ?struct drm_sched_job *panthor_job_get(struct drm_sched_job *job); >> ?struct panthor_vm *panthor_job_vm(struct drm_sched_job *sched_job); >> ?void panthor_job_put(struct drm_sched_job *job); >> diff --git a/drivers/gpu/drm/scheduler/sched_fence.c >> b/drivers/gpu/drm/scheduler/sched_fence.c >> index e971528504a5..d208d384d38d 100644 >> --- a/drivers/gpu/drm/scheduler/sched_fence.c >> +++ b/drivers/gpu/drm/scheduler/sched_fence.c >> @@ -206,7 +206,8 @@ struct drm_sched_fence *to_drm_sched_fence(struct >> dma_fence *f) >> ?EXPORT_SYMBOL(to_drm_sched_fence); >> >> ?struct drm_sched_fence *drm_sched_fence_alloc(struct >> drm_sched_entity *entity, >> - ????? void *owner) >> + ????? void *owner, >> + ????? u64 drm_client_id) >> ?{ >> ? struct drm_sched_fence *fence = NULL; >> >> @@ -215,6 +216,7 @@ struct drm_sched_fence >> *drm_sched_fence_alloc(struct drm_sched_entity *entity, >> ? return NULL; >> >> ? fence->owner = owner; >> + fence->drm_client_id = drm_client_id; >> ? spin_lock_init(&fence->lock); >> >> ? return fence; >> diff --git a/drivers/gpu/drm/scheduler/sched_internal.h >> b/drivers/gpu/drm/scheduler/sched_internal.h >> index 599cf6e1bb74..7ea5a6736f98 100644 >> --- a/drivers/gpu/drm/scheduler/sched_internal.h >> +++ b/drivers/gpu/drm/scheduler/sched_internal.h >> @@ -24,7 +24,7 @@ void drm_sched_entity_select_rq(struct >> drm_sched_entity *entity); >> ?struct drm_sched_job *drm_sched_entity_pop_job(struct >> drm_sched_entity *entity); >> >> ?struct drm_sched_fence *drm_sched_fence_alloc(struct >> drm_sched_entity *s_entity, >> - ????? void *owner); >> + ????? void *owner, u64 >> drm_client_id); >> ?void drm_sched_fence_init(struct drm_sched_fence *fence, >> ? ? struct drm_sched_entity *entity); >> ?void drm_sched_fence_free(struct drm_sched_fence *fence); >> diff --git a/drivers/gpu/drm/scheduler/sched_main.c >> b/drivers/gpu/drm/scheduler/sched_main.c >> index 829579c41c6b..60611618f3ab 100644 >> --- a/drivers/gpu/drm/scheduler/sched_main.c >> +++ b/drivers/gpu/drm/scheduler/sched_main.c >> @@ -764,6 +764,7 @@ EXPORT_SYMBOL(drm_sched_resubmit_jobs); >> ? * @credits: the number of credits this job contributes to the >> schedulers >> ? * credit limit >> ? * @owner: job owner for debugging >> + * @drm_client_id: drm_file.client_id of the owner > > For the docu generation to link that properly it must be written as > > &struct drm_file.client_id Noted. > > Besides, if this were an optional parameter, one should document it. > I'm not sure if it is, I haven't used these client_id's before. Passing an invalid client_id would only cause the trace events to print the invalid client_id. Thanks, Pierre-Eric > > P. > >> ? * >> ? * Refer to drm_sched_entity_push_job() documentation >> ? * for locking considerations. >> @@ -784,7 +785,8 @@ EXPORT_SYMBOL(drm_sched_resubmit_jobs); >> ? */ >> ?int drm_sched_job_init(struct drm_sched_job *job, >> ? ?????? struct drm_sched_entity *entity, >> - ?????? u32 credits, void *owner) >> + ?????? u32 credits, void *owner, >> + ?????? uint64_t drm_client_id) >> ?{ >> ? if (!entity->rq) { >> ? /* This will most likely be followed by missing >> frames >> @@ -810,7 +812,7 @@ int drm_sched_job_init(struct drm_sched_job *job, >> >> ? job->entity = entity; >> ? job->credits = credits; >> - job->s_fence = drm_sched_fence_alloc(entity, owner); >> + job->s_fence = drm_sched_fence_alloc(entity, owner, >> drm_client_id); >> ? if (!job->s_fence) >> ? return -ENOMEM; >> >> diff --git a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c >> b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c >> index f999c8859cf7..09ffbdb32d76 100644 >> --- a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c >> +++ b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c >> @@ -35,7 +35,7 @@ drm_mock_sched_entity_new(struct kunit *test, >> ? ret = drm_sched_entity_init(&entity->base, >> ? ??? priority, >> ? ??? &drm_sched, 1, >> - ??? NULL); >> + ??? NULL, 1); >> ? KUNIT_ASSERT_EQ(test, ret, 0); >> >> ? entity->test = test; >> diff --git a/drivers/gpu/drm/v3d/v3d_submit.c >> b/drivers/gpu/drm/v3d/v3d_submit.c >> index 4ff5de46fb22..5171ffe9012d 100644 >> --- a/drivers/gpu/drm/v3d/v3d_submit.c >> +++ b/drivers/gpu/drm/v3d/v3d_submit.c >> @@ -169,7 +169,7 @@ v3d_job_init(struct v3d_dev *v3d, struct drm_file >> *file_priv, >> ? job->file = file_priv; >> >> ? ret = drm_sched_job_init(&job->base, &v3d_priv- >>> sched_entity[queue], >> - 1, v3d_priv); >> + 1, v3d_priv, file_priv->client_id); >> ? if (ret) >> ? return ret; >> >> diff --git a/drivers/gpu/drm/xe/xe_sched_job.c >> b/drivers/gpu/drm/xe/xe_sched_job.c >> index 1905ca590965..f4679cb9a56b 100644 >> --- a/drivers/gpu/drm/xe/xe_sched_job.c >> +++ b/drivers/gpu/drm/xe/xe_sched_job.c >> @@ -113,7 +113,8 @@ struct xe_sched_job *xe_sched_job_create(struct >> xe_exec_queue *q, >> ? kref_init(&job->refcount); >> ? xe_exec_queue_get(job->q); >> >> - err = drm_sched_job_init(&job->drm, q->entity, 1, NULL); >> + err = drm_sched_job_init(&job->drm, q->entity, 1, NULL, >> + q->xef->drm->client_id); >> ? if (err) >> ? goto err_free; >> >> diff --git a/include/drm/gpu_scheduler.h >> b/include/drm/gpu_scheduler.h >> index 1a7e377d4cbb..6fe3b4c0cffb 100644 >> --- a/include/drm/gpu_scheduler.h >> +++ b/include/drm/gpu_scheduler.h >> @@ -305,6 +305,13 @@ struct drm_sched_fence { >> ????????? * @owner: job owner for debugging >> ????????? */ >> ? void *owner; >> + >> + /** >> + * @drm_client_id: >> + * >> + * The client_id of the drm_file which owns the job. >> + */ >> + uint64_t drm_client_id; >> ?}; >> >> ?struct drm_sched_fence *to_drm_sched_fence(struct dma_fence *f); >> @@ -629,7 +636,8 @@ drm_sched_pick_best(struct drm_gpu_scheduler >> **sched_list, >> >> ?int drm_sched_job_init(struct drm_sched_job *job, >> ? ?????? struct drm_sched_entity *entity, >> - ?????? u32 credits, void *owner); >> + ?????? u32 credits, void *owner, >> + ?????? u64 drm_client_id); >> ?void drm_sched_job_arm(struct drm_sched_job *job); >> ?void drm_sched_entity_push_job(struct drm_sched_job *sched_job); >> ?int drm_sched_job_add_dependency(struct drm_sched_job *job, From mark.filion at collabora.com Sat May 17 09:44:50 2025 From: mark.filion at collabora.com (Mark Filion) Date: Sat, 17 May 2025 11:44:50 +0200 Subject: 2025 X.Org Foundation Election results Message-ID: <17f8e9490268bfd6c87960ef03a571a05364bf29.camel@collabora.com> Hello all, The Board of Directors election concluded at 23:59 UTC on 14 May 2025. For this election, we had 77 members who could cast a vote. 72 did, so the turnout was 92.5%. This is the 3rd highest turnout in the last 10 elections, with only 2021 (93.8%) and 2016 (98.8%) being higher. It is also a 147% increase from last year's turnout (62.9%). Thank you to everyone who took the time to vote! In the election of the Directors to the Board of the X.Org Foundation, the results were that Lyude Paul, Andres Gomez, Arkadiusz Hiler and Harry Wentland were elected for two year terms. The old full board is: Erik Faye-Lund, Mark Filion, Neal Gompa, Arkadiusz Hiler, Christopher Michael, Lyude Paul, and Simon Ser. The new full board is: Erik Faye-Lund, Mark Filion, Andres Gomez, Neal Gompa, Arkadiusz Hiler, Lyude Paul, Simon Ser, and Harry Wentland. Mark Filion, on behalf of the X.Org elections committee -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre-eric at damsy.net Mon May 19 11:02:45 2025 From: pierre-eric at damsy.net (Pierre-Eric Pelloux-Prayer) Date: Mon, 19 May 2025 13:02:45 +0200 Subject: [PATCH v9 02/10] drm/sched: store the drm client_id in drm_sched_fence In-Reply-To: <2d206814-968d-47df-b670-cbd8db68c5c6@damsy.net> References: <20250424083834.15518-1-pierre-eric.pelloux-prayer@amd.com> <20250424083834.15518-3-pierre-eric.pelloux-prayer@amd.com> <2d206814-968d-47df-b670-cbd8db68c5c6@damsy.net> Message-ID: <882eeee1-cc36-4287-8437-b39c6c84562e@damsy.net> Le 15/05/2025 ? 08:53, Pierre-Eric Pelloux-Prayer a ?crit?: > Hi, > > Le 14/05/2025 ? 14:44, Philipp Stanner a ?crit?: >> On Thu, 2025-04-24 at 10:38 +0200, Pierre-Eric Pelloux-Prayer wrote: >>> This will be used in a later commit to trace the drm client_id in >>> some of the gpu_scheduler trace events. >>> >>> This requires changing all the users of drm_sched_job_init to >>> add an extra parameter. >>> >>> The newly added drm_client_id field in the drm_sched_fence is a bit >>> of a duplicate of the owner one. One suggestion I received was to >>> merge those 2 fields - this can't be done right now as amdgpu uses >>> some special values (AMDGPU_FENCE_OWNER_*) that can't really be >>> translated into a client id. Christian is working on getting rid of >>> those; when it's done we should be able to squash owner/drm_client_id >>> together. >>> >>> Reviewed-by: Christian K?nig >>> Signed-off-by: Pierre-Eric Pelloux-Prayer >>> >>> --- >>> ??drivers/accel/amdxdna/aie2_ctx.c???????????????? |? 3 ++- >>> ??drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c?????? |? 2 +- >>> ??drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c?????????? |? 3 ++- >>> ??drivers/gpu/drm/amd/amdgpu/amdgpu_job.c????????? |? 8 +++++--- >>> ??drivers/gpu/drm/amd/amdgpu/amdgpu_job.h????????? |? 3 ++- >>> ??drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c???? |? 2 +- >>> ??drivers/gpu/drm/imagination/pvr_job.c??????????? |? 2 +- >>> ??drivers/gpu/drm/imagination/pvr_queue.c????????? |? 5 +++-- >>> ??drivers/gpu/drm/imagination/paac???????????? |? 8 +++++--- >>> ??drivers/gpu/drm/nouveau/nouveau_sched.c????????? |? 3 ++- >>> ??drivers/gpu/drm/panfrost/panfrost_drv.c????????? |? 2 +- >>> ??drivers/gpu/drm/panthor/panthor_drv.c??????????? |? 3 ++- >>> ??drivers/gpu/drm/panthor/panthor_mmu.c??????????? |? 2 +- >>> ??drivers/gpu/drm/panthor/panthor_sched.c????????? |? 5 +++-- >>> ??drivers/gpu/drm/panthor/panthor_sched.h????????? |? 3 ++- >>> ??drivers/gpu/drm/scheduler/sched_fence.c????????? |? 4 ++ >>> ??drivers/gpu/drm/scheduler/sched_internal.h?????? |? 2 +- >>> ??drivers/gpu/drm/scheduler/sched_main.c?????????? |? 6 ++++-- >>> ??drivers/gpu/drm/scheduler/tests/mock_scheduler.c |? 2 +- >>> ??drivers/gpu/drm/v3d/v3d_submit.c???????????????? |? 2 +- >>> ??drivers/gpu/drm/xe/xe_sched_job.c??????????????? |? 3 ++- >>> ??include/drm/gpu_scheduler.h????????????????????? | 10 +++++++++- >>> ??26 files changed, 62 insertions(+), 34 deletions(-) >> >> I think last time I asked about what your merge plan for this is, since >> it touches so many drivers. Should I take that? > > Based on: > > https://drm.pages.freedesktop.org/maintainer-tools/committer/committer-drm-misc.html > > "drm-misc is for drm core (non-driver) patches, subsystem-wide refactorings, > and small trivial patches all over (including drivers)." > > I assume it should go through drm-misc. I've addressed your comments and pushed an updated branch to https://gitlab.freedesktop.org/pepp/linux/-/commits/improve_gpu_scheduler_trace_v10 Any chance to get this merged soon? Thanks, Pierre-Eric > > >> >> Besides one comment below, scheduler bits look fine. >> >>> >>> diff --git a/drivers/accel/amdxdna/aie2_ctx.c >>> b/drivers/accel/amdxdna/aie2_ctx.c >>> index e04549f64d69..3e38a5f637ea 100644 >>> --- a/drivers/accel/amdxdna/aie2_ctx.c >>> +++ b/drivers/accel/amdxdna/aie2_ctx.c >>> @@ -848,7 +848,8 @@ int aie2_cmd_submit(struct amdxdna_hwctx *hwctx, >>> struct amdxdna_sched_job *job, >>> ????????? goto up_sem; >>> ????? } >>> -??? ret = drm_sched_job_init(&job->base, &hwctx->priv->entity, >>> 1, hwctx); >>> +??? ret = drm_sched_job_init(&job->base, &hwctx->priv->entity, >>> 1, hwctx, >>> +???????????????? hwctx->client->filp->client_id); >>> ????? if (ret) { >>> ????????? XDNA_ERR(xdna, "DRM job init failed, ret %d", ret); >>> ????????? goto free_chain; >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c >>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c >>> index 4cec3a873995..1a77ba7036c9 100644 >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c >>> @@ -639,7 +639,7 @@ int amdgpu_amdkfd_submit_ib(struct amdgpu_device >>> *adev, >>> ????????? goto err; >>> ????? } >>> -??? ret = amdgpu_job_alloc(adev, NULL, NULL, NULL, 1, &job); >>> +??? ret = amdgpu_job_alloc(adev, NULL, NULL, NULL, 1, &job, 0); >>> ????? if (ret) >>> ????????? goto err; >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c >>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c >>> index 82df06a72ee0..5a231b997d65 100644 >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c >>> @@ -293,7 +293,8 @@ static int amdgpu_cs_pass1(struct >>> amdgpu_cs_parser *p, >>> ????? for (i = 0; i < p->gang_size; ++i) { >>> ????????? ret = amdgpu_job_alloc(p->adev, vm, p->entities[i], >>> vm, >>> -??????????????? ?????? num_ibs[i], &p->jobs[i]); >>> +??????????????? ?????? num_ibs[i], &p->jobs[i], >>> +??????????????? ?????? p->filp->client_id); >>> ????????? if (ret) >>> ????????????? goto free_all_kdata; >>> ????????? p->jobs[i]->enforce_isolation = p->adev- >>>> enforce_isolation[fpriv->xcp_id]; >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c >>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c >>> index acb21fc8b3ce..75262ce8db27 100644 >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c >>> @@ -204,7 +204,8 @@ static enum drm_gpu_sched_stat >>> amdgpu_job_timedout(struct drm_sched_job *s_job) >>> ??int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm >>> *vm, >>> ????????? ???? struct drm_sched_entity *entity, void *owner, >>> -??????? ???? unsigned int num_ibs, struct amdgpu_job **job) >>> +??????? ???? unsigned int num_ibs, struct amdgpu_job **job, >>> +??????? ???? u64 drm_client_id) >>> ??{ >>> ????? if (num_ibs == 0) >>> ????????? return -EINVAL; >>> @@ -222,7 +223,8 @@ int amdgpu_job_alloc(struct amdgpu_device *adev, >>> struct amdgpu_vm *vm, >>> ????? if (!entity) >>> ????????? return 0; >>> -??? return drm_sched_job_init(&(*job)->base, entity, 1, owner); >>> +??? return drm_sched_job_init(&(*job)->base, entity, 1, owner, >>> +??????????????? ? drm_client_id); >>> ??} >>> ??int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, >>> @@ -232,7 +234,7 @@ int amdgpu_job_alloc_with_ib(struct amdgpu_device >>> *adev, >>> ??{ >>> ????? int r; >>> -??? r = amdgpu_job_alloc(adev, NULL, entity, owner, 1, job); >>> +??? r = amdgpu_job_alloc(adev, NULL, entity, owner, 1, job, 0); >>> ????? if (r) >>> ????????? return r; >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h >>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h >>> index ce6b9ba967ff..5a8bc6342222 100644 >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h >>> @@ -90,7 +90,8 @@ static inline struct amdgpu_ring >>> *amdgpu_job_ring(struct amdgpu_job *job) >>> ??int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm >>> *vm, >>> ????????? ???? struct drm_sched_entity *entity, void *owner, >>> -??????? ???? unsigned int num_ibs, struct amdgpu_job **job); >>> +??????? ???? unsigned int num_ibs, struct amdgpu_job **job, >>> +??????? ???? u64 drm_client_id); >>> ??int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, >>> ????????????? ???? struct drm_sched_entity *entity, void >>> *owner, >>> ????????????? ???? size_t size, enum amdgpu_ib_pool_type >>> pool_type, >>> diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c >>> b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c >>> index 3c0a5c3e0e3d..76c742328edb 100644 >>> --- a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c >>> +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c >>> @@ -534,7 +534,7 @@ int etnaviv_ioctl_gem_submit(struct drm_device >>> *dev, void *data, >>> ????? ret = drm_sched_job_init(&submit->sched_job, >>> ?????????????????? &ctx->sched_entity[args->pipe], >>> -???????????????? 1, submit->ctx); >>> +???????????????? 1, submit->ctx, file->client_id); >>> ????? if (ret) >>> ????????? goto err_submit_put; >>> diff --git a/drivers/gpu/drm/imagination/pvr_job.c >>> b/drivers/gpu/drm/imagination/pvr_job.c >>> index 59b334d094fa..7564b0f21b42 100644 >>> --- a/drivers/gpu/drm/imagination/pvr_job.c >>> +++ b/drivers/gpu/drm/imagination/pvr_job.c >>> @@ -446,7 +446,7 @@ create_job(struct pvr_device *pvr_dev, >>> ????? if (err) >>> ????????? goto err_put_job; >>> -??? err = pvr_queue_job_init(job); >>> +??? err = pvr_queue_job_init(job, pvr_file->file->client_id); >>> ????? if (err) >>> ????????? goto err_put_job; >>> diff --git a/drivers/gpu/drm/imagination/pvr_queue.c >>> b/drivers/gpu/drm/imagination/pvr_queue.c >>> index 5e9bc0992824..5a41ee79fed6 100644 >>> --- a/drivers/gpu/drm/imagination/pvr_queue.c >>> +++ b/drivers/gpu/drm/imagination/pvr_queue.c >>> @@ -1073,6 +1073,7 @@ static int pvr_queue_cleanup_fw_context(struct >>> pvr_queue *queue) >>> ??/** >>> ?? * pvr_queue_job_init() - Initialize queue related fields in a >>> pvr_job object. >>> ?? * @job: The job to initialize. >>> + * @drm_client_id: drm_file.client_id submitting the job >>> ?? * >>> ?? * Bind the job to a queue and allocate memory to guarantee >>> pvr_queue_job_arm() >>> ?? * and pvr_queue_job_push() can't fail. We also make sure the >>> context type is >>> @@ -1082,7 +1083,7 @@ static int pvr_queue_cleanup_fw_context(struct >>> pvr_queue *queue) >>> ?? *? * 0 on success, or >>> ?? *? * An error code if something failed. >>> ?? */ >>> -int pvr_queue_job_init(struct pvr_job *job) >>> +int pvr_queue_job_init(struct pvr_job *job, u64 drm_client_id) >>> ??{ >>> ????? /* Fragment jobs need at least one native fence wait on the >>> geometry job fence. */ >>> ????? u32 min_native_dep_count = job->type == >>> DRM_PVR_JOB_TYPE_FRAGMENT ? 1 : 0; >>> @@ -1099,7 +1100,7 @@ int pvr_queue_job_init(struct pvr_job *job) >>> ????? if (!pvr_cccb_cmdseq_can_fit(&queue->cccb, >>> job_cmds_size(job, min_native_dep_count))) >>> ????????? return -E2BIG; >>> -??? err = drm_sched_job_init(&job->base, &queue->entity, 1, >>> THIS_MODULE); >>> +??? err = drm_sched_job_init(&job->base, &queue->entity, 1, >>> THIS_MODULE, drm_client_id); >>> ????? if (err) >>> ????????? return err; >>> diff --git a/drivers/gpu/drm/imagination/pvr_queue.h >>> b/drivers/gpu/drm/imagination/pvr_queue.h >>> index 93fe9ac9f58c..fc1986d73fc8 100644 >>> --- a/drivers/gpu/drm/imagination/pvr_queue.h >>> +++ b/drivers/gpu/drm/imagination/pvr_queue.h >>> @@ -143,7 +143,7 @@ struct pvr_queue { >>> ??bool pvr_queue_fence_is_ufo_backed(struct dma_fence *f); >>> -int pvr_queue_job_init(struct pvr_job *job); >>> +int pvr_queue_job_init(struct pvr_job *job, u64 drm_client_id); >>> ??void pvr_queue_job_cleanup(struct pvr_job *job); >>> diff --git a/drivers/gpu/drm/lima/lima_gem.c >>> b/drivers/gpu/drm/lima/lima_gem.c >>> index 5deec673c11e..9722b847a539 100644 >>> --- a/drivers/gpu/drm/lima/lima_gem.c >>> +++ b/drivers/gpu/drm/lima/lima_gem.c >>> @@ -341,7 +341,7 @@ int lima_gem_submit(struct drm_file *file, struct >>> lima_submit *submit) >>> ????? err = lima_sched_task_init( >>> ????????? submit->task, submit->ctx->context + submit->pipe, >>> -??????? bos, submit->nr_bos, vm); >>> +??????? bos, submit->nr_bos, vm, file->client_id); >>> ????? if (err) >>> ????????? goto err_out1; >>> diff --git a/drivers/gpu/drm/lima/lima_sched.c >>> b/drivers/gpu/drm/lima/lima_sched.c >>> index 7934098e651b..954f4325b859 100644 >>> --- a/drivers/gpu/drm/lima/lima_sched.c >>> +++ b/drivers/gpu/drm/lima/lima_sched.c >>> @@ -113,7 +113,8 @@ static inline struct lima_sched_pipe >>> *to_lima_pipe(struct drm_gpu_scheduler *sch >>> ??int lima_sched_task_init(struct lima_sched_task *task, >>> ?????????????? struct lima_sched_context *context, >>> ?????????????? struct lima_bo **bos, int num_bos, >>> -???????????? struct lima_vm *vm) >>> +???????????? struct lima_vm *vm, >>> +???????????? u64 drm_client_id) >>> ??{ >>> ????? int err, i; >>> @@ -124,7 +125,8 @@ int lima_sched_task_init(struct lima_sched_task >>> *task, >>> ????? for (i = 0; i < num_bos; i++) >>> ????????? drm_gem_object_get(&bos[i]->base.base); >>> -??? err = drm_sched_job_init(&task->base, &context->base, 1, >>> vm); >>> +??? err = drm_sched_job_init(&task->base, &context->base, 1, vm, >>> +???????????????? drm_client_id); >>> ????? if (err) { >>> ????????? kfree(task->bos); >>> ????????? return err; >>> diff --git a/drivers/gpu/drm/lima/lima_sched.h >>> b/drivers/gpu/drm/lima/lima_sched.h >>> index 85b23ba901d5..1a08faf8a529 100644 >>> --- a/drivers/gpu/drm/lima/lima_sched.h >>> +++ b/drivers/gpu/drm/lima/lima_sched.h >>> @@ -88,7 +88,8 @@ struct lima_sched_pipe { >>> ??int lima_sched_task_init(struct lima_sched_task *task, >>> ?????????????? struct lima_sched_context *context, >>> ?????????????? struct lima_bo **bos, int num_bos, >>> -???????????? struct lima_vm *vm); >>> +???????????? struct lima_vm *vm, >>> +???????????? u64 drm_client_id); >>> ??void lima_sched_task_fini(struct lima_sched_task *task); >>> ??int lima_sched_context_init(struct lima_sched_pipe *pipe, >>> diff --git a/drivers/gpu/drm/msm/msm_gem_submit.c >>> b/drivers/gpu/drm/msm/msm_gem_submit.c >>> index 3e9aa2cc38ef..d9be0fe3d674 100644 >>> --- a/drivers/gpu/drm/msm/msm_gem_submit.c >>> +++ b/drivers/gpu/drm/msm/msm_gem_submit.c >>> @@ -30,7 +30,7 @@ >>> ??static struct msm_gem_submit *submit_create(struct drm_device *dev, >>> ????????? struct msm_gpu *gpu, >>> ????????? struct msm_gpu_submitqueue *queue, uint32_t nr_bos, >>> -??????? uint32_t nr_cmds) >>> +??????? uint32_t nr_cmds, u64 drm_client_id) >>> ??{ >>> ????? static atomic_t ident = ATOMIC_INIT(0); >>> ????? struct msm_gem_submit *submit; >>> @@ -54,7 +54,8 @@ static struct msm_gem_submit *submit_create(struct >>> drm_device *dev, >>> ????????? return ERR_PTR(ret); >>> ????? } >>> -??? ret = drm_sched_job_init(&submit->base, queue->entity, 1, >>> queue); >>> +??? ret = drm_sched_job_init(&submit->base, queue->entity, 1, >>> queue, >>> +???????????????? drm_client_id); >>> ????? if (ret) { >>> ????????? kfree(submit->hw_fence); >>> ????????? kfree(submit); >>> @@ -693,7 +694,8 @@ int msm_ioctl_gem_submit(struct drm_device *dev, >>> void *data, >>> ????????? } >>> ????? } >>> -??? submit = submit_create(dev, gpu, queue, args->nr_bos, args- >>>> nr_cmds); >>> +??? submit = submit_create(dev, gpu, queue, args->nr_bos, args- >>>> nr_cmds, >>> +??????????? ?????? file->client_id); >>> ????? if (IS_ERR(submit)) { >>> ????????? ret = PTR_ERR(submit); >>> ????????? goto out_post_unlock; >>> diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.c >>> b/drivers/gpu/drm/nouveau/nouveau_sched.c >>> index d326e55d2d24..460a5fb02412 100644 >>> --- a/drivers/gpu/drm/nouveau/nouveau_sched.c >>> +++ b/drivers/gpu/drm/nouveau/nouveau_sched.c >>> @@ -87,7 +87,8 @@ nouveau_job_init(struct nouveau_job *job, >>> ????? } >>> ????? ret = drm_sched_job_init(&job->base, &sched->entity, >>> -???????????????? args->credits, NULL); >>> +???????????????? args->credits, NULL, >>> +???????????????? job->file_priv->client_id); >>> ????? if (ret) >>> ????????? goto err_free_chains; >>> diff --git a/drivers/gpu/drm/panfrost/panfrost_drv.c >>> b/drivers/gpu/drm/panfrost/panfrost_drv.c >>> index b87f83e94eda..d5c2c6530ed8 100644 >>> --- a/drivers/gpu/drm/panfrost/panfrost_drv.c >>> +++ b/drivers/gpu/drm/panfrost/panfrost_drv.c >>> @@ -312,7 +312,7 @@ static int panfrost_ioctl_submit(struct >>> drm_device *dev, void *data, >>> ????? ret = drm_sched_job_init(&job->base, >>> ?????????????????? &file_priv->sched_entity[slot], >>> -???????????????? 1, NULL); >>> +???????????????? 1, NULL, file->client_id); >>> ????? if (ret) >>> ????????? goto out_put_job; >>> diff --git a/drivers/gpu/drm/panthor/panthor_drv.c >>> b/drivers/gpu/drm/panthor/panthor_drv.c >>> index 06fe46e32073..bd8e1900c919 100644 >>> --- a/drivers/gpu/drm/panthor/panthor_drv.c >>> +++ b/drivers/gpu/drm/panthor/panthor_drv.c >>> @@ -989,7 +989,8 @@ static int panthor_ioctl_group_submit(struct >>> drm_device *ddev, void *data, >>> ????????? const struct drm_panthor_queue_submit *qsubmit = >>> &jobs_args[i]; >>> ????????? struct drm_sched_job *job; >>> -??????? job = panthor_job_create(pfile, args->group_handle, >>> qsubmit); >>> +??????? job = panthor_job_create(pfile, args->group_handle, >>> qsubmit, >>> +???????????????????? file->client_id); >>> ????????? if (IS_ERR(job)) { >>> ????????????? ret = PTR_ERR(job); >>> ????????????? goto out_cleanup_submit_ctx; >>> diff --git a/drivers/gpu/drm/panthor/panthor_mmu.c >>> b/drivers/gpu/drm/panthor/panthor_mmu.c >>> index 12a02e28f50f..e0c79bd2d173 100644 >>> --- a/drivers/gpu/drm/panthor/panthor_mmu.c >>> +++ b/drivers/gpu/drm/panthor/panthor_mmu.c >>> @@ -2516,7 +2516,7 @@ panthor_vm_bind_job_create(struct drm_file >>> *file, >>> ????? kref_init(&job->refcount); >>> ????? job->vm = panthor_vm_get(vm); >>> -??? ret = drm_sched_job_init(&job->base, &vm->entity, 1, vm); >>> +??? ret = drm_sched_job_init(&job->base, &vm->entity, 1, vm, >>> file->client_id); >>> ????? if (ret) >>> ????????? goto err_put_job; >>> diff --git a/drivers/gpu/drm/panthor/panthor_sched.c >>> b/drivers/gpu/drm/panthor/panthor_sched.c >>> index 446ec780eb4a..2af860c9068a 100644 >>> --- a/drivers/gpu/drm/panthor/panthor_sched.c >>> +++ b/drivers/gpu/drm/panthor/panthor_sched.c >>> @@ -3729,7 +3729,8 @@ struct panthor_vm *panthor_job_vm(struct >>> drm_sched_job *sched_job) >>> ??struct drm_sched_job * >>> ??panthor_job_create(struct panthor_file *pfile, >>> ????????? ?? u16 group_handle, >>> -??????? ?? const struct drm_panthor_queue_submit *qsubmit) >>> +??????? ?? const struct drm_panthor_queue_submit *qsubmit, >>> +??????? ?? u64 drm_client_id) >>> ??{ >>> ????? struct panthor_group_pool *gpool = pfile->groups; >>> ????? struct panthor_job *job; >>> @@ -3801,7 +3802,7 @@ panthor_job_create(struct panthor_file *pfile, >>> ????? ret = drm_sched_job_init(&job->base, >>> ?????????????????? &job->group->queues[job- >>>> queue_idx]->entity, >>> -???????????????? credits, job->group); >>> +???????????????? credits, job->group, >>> drm_client_id); >>> ????? if (ret) >>> ????????? goto err_put_job; >>> diff --git a/drivers/gpu/drm/panthor/panthor_sched.h >>> b/drivers/gpu/drm/panthor/panthor_sched.h >>> index e650a445cf50..742b0b4ff3a3 100644 >>> --- a/drivers/gpu/drm/panthor/panthor_sched.h >>> +++ b/drivers/gpu/drm/panthor/panthor_sched.h >>> @@ -29,7 +29,8 @@ int panthor_group_get_state(struct panthor_file >>> *pfile, >>> ??struct drm_sched_job * >>> ??panthor_job_create(struct panthor_file *pfile, >>> ????????? ?? u16 group_handle, >>> -??????? ?? const struct drm_panthor_queue_submit *qsubmit); >>> +??????? ?? const struct drm_panthor_queue_submit *qsubmit, >>> +??????? ?? u64 drm_client_id); >>> ??struct drm_sched_job *panthor_job_get(struct drm_sched_job *job); >>> ??struct panthor_vm *panthor_job_vm(struct drm_sched_job *sched_job); >>> ??void panthor_job_put(struct drm_sched_job *job); >>> diff --git a/drivers/gpu/drm/scheduler/sched_fence.c >>> b/drivers/gpu/drm/scheduler/sched_fence.c >>> index e971528504a5..d208d384d38d 100644 >>> --- a/drivers/gpu/drm/scheduler/sched_fence.c >>> +++ b/drivers/gpu/drm/scheduler/sched_fence.c >>> @@ -206,7 +206,8 @@ struct drm_sched_fence *to_drm_sched_fence(struct >>> dma_fence *f) >>> ??EXPORT_SYMBOL(to_drm_sched_fence); >>> ??struct drm_sched_fence *drm_sched_fence_alloc(struct >>> drm_sched_entity *entity, >>> -??????????????????? ????? void *owner) >>> +??????????????????? ????? void *owner, >>> +??????????????????? ????? u64 drm_client_id) >>> ??{ >>> ????? struct drm_sched_fence *fence = NULL; >>> @@ -215,6 +216,7 @@ struct drm_sched_fence >>> *drm_sched_fence_alloc(struct drm_sched_entity *entity, >>> ????????? return NULL; >>> ????? fence->owner = owner; >>> +??? fence->drm_client_id = drm_client_id; >>> ????? spin_lock_init(&fence->lock); >>> ????? return fence; >>> diff --git a/drivers/gpu/drm/scheduler/sched_internal.h >>> b/drivers/gpu/drm/scheduler/sched_internal.h >>> index 599cf6e1bb74..7ea5a6736f98 100644 >>> --- a/drivers/gpu/drm/scheduler/sched_internal.h >>> +++ b/drivers/gpu/drm/scheduler/sched_internal.h >>> @@ -24,7 +24,7 @@ void drm_sched_entity_select_rq(struct >>> drm_sched_entity *entity); >>> ??struct drm_sched_job *drm_sched_entity_pop_job(struct >>> drm_sched_entity *entity); >>> ??struct drm_sched_fence *drm_sched_fence_alloc(struct >>> drm_sched_entity *s_entity, >>> -??????????????????? ????? void *owner); >>> +??????????????????? ????? void *owner, u64 >>> drm_client_id); >>> ??void drm_sched_fence_init(struct drm_sched_fence *fence, >>> ????????????? ? struct drm_sched_entity *entity); >>> ??void drm_sched_fence_free(struct drm_sched_fence *fence); >>> diff --git a/drivers/gpu/drm/scheduler/sched_main.c >>> b/drivers/gpu/drm/scheduler/sched_main.c >>> index 829579c41c6b..60611618f3ab 100644 >>> --- a/drivers/gpu/drm/scheduler/sched_main.c >>> +++ b/drivers/gpu/drm/scheduler/sched_main.c >>> @@ -764,6 +764,7 @@ EXPORT_SYMBOL(drm_sched_resubmit_jobs); >>> ?? * @credits: the number of credits this job contributes to the >>> schedulers >>> ?? * credit limit >>> ?? * @owner: job owner for debugging >>> + * @drm_client_id: drm_file.client_id of the owner >> >> For the docu generation to link that properly it must be written as >> >> &struct drm_file.client_id > > Noted. > >> >> Besides, if this were an optional parameter, one should document it. >> I'm not sure if it is, I haven't used these client_id's before. > > Passing an invalid client_id would only cause the trace events to print the invalid client_id. > > Thanks, > Pierre-Eric > > >> >> P. >> >>> ?? * >>> ?? * Refer to drm_sched_entity_push_job() documentation >>> ?? * for locking considerations. >>> @@ -784,7 +785,8 @@ EXPORT_SYMBOL(drm_sched_resubmit_jobs); >>> ?? */ >>> ??int drm_sched_job_init(struct drm_sched_job *job, >>> ????????? ?????? struct drm_sched_entity *entity, >>> -??????? ?????? u32 credits, void *owner) >>> +??????? ?????? u32 credits, void *owner, >>> +??????? ?????? uint64_t drm_client_id) >>> ??{ >>> ????? if (!entity->rq) { >>> ????????? /* This will most likely be followed by missing >>> frames >>> @@ -810,7 +812,7 @@ int drm_sched_job_init(struct drm_sched_job *job, >>> ????? job->entity = entity; >>> ????? job->credits = credits; >>> -??? job->s_fence = drm_sched_fence_alloc(entity, owner); >>> +??? job->s_fence = drm_sched_fence_alloc(entity, owner, >>> drm_client_id); >>> ????? if (!job->s_fence) >>> ????????? return -ENOMEM; >>> diff --git a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c >>> b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c >>> index f999c8859cf7..09ffbdb32d76 100644 >>> --- a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c >>> +++ b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c >>> @@ -35,7 +35,7 @@ drm_mock_sched_entity_new(struct kunit *test, >>> ????? ret = drm_sched_entity_init(&entity->base, >>> ????????????????? ??? priority, >>> ????????????????? ??? &drm_sched, 1, >>> -??????????????? ??? NULL); >>> +??????????????? ??? NULL, 1); >>> ????? KUNIT_ASSERT_EQ(test, ret, 0); >>> ????? entity->test = test; >>> diff --git a/drivers/gpu/drm/v3d/v3d_submit.c >>> b/drivers/gpu/drm/v3d/v3d_submit.c >>> index 4ff5de46fb22..5171ffe9012d 100644 >>> --- a/drivers/gpu/drm/v3d/v3d_submit.c >>> +++ b/drivers/gpu/drm/v3d/v3d_submit.c >>> @@ -169,7 +169,7 @@ v3d_job_init(struct v3d_dev *v3d, struct drm_file >>> *file_priv, >>> ????? job->file = file_priv; >>> ????? ret = drm_sched_job_init(&job->base, &v3d_priv- >>>> sched_entity[queue], >>> -???????????????? 1, v3d_priv); >>> +???????????????? 1, v3d_priv, file_priv->client_id); >>> ????? if (ret) >>> ????????? return ret; >>> diff --git a/drivers/gpu/drm/xe/xe_sched_job.c >>> b/drivers/gpu/drm/xe/xe_sched_job.c >>> index 1905ca590965..f4679cb9a56b 100644 >>> --- a/drivers/gpu/drm/xe/xe_sched_job.c >>> +++ b/drivers/gpu/drm/xe/xe_sched_job.c >>> @@ -113,7 +113,8 @@ struct xe_sched_job *xe_sched_job_create(struct >>> xe_exec_queue *q, >>> ????? kref_init(&job->refcount); >>> ????? xe_exec_queue_get(job->q); >>> -??? err = drm_sched_job_init(&job->drm, q->entity, 1, NULL); >>> +??? err = drm_sched_job_init(&job->drm, q->entity, 1, NULL, >>> +???????????????? q->xef->drm->client_id); >>> ????? if (err) >>> ????????? goto err_free; >>> diff --git a/include/drm/gpu_scheduler.h >>> b/include/drm/gpu_scheduler.h >>> index 1a7e377d4cbb..6fe3b4c0cffb 100644 >>> --- a/include/drm/gpu_scheduler.h >>> +++ b/include/drm/gpu_scheduler.h >>> @@ -305,6 +305,13 @@ struct drm_sched_fence { >>> ?????????? * @owner: job owner for debugging >>> ?????????? */ >>> ????? void??????????????? *owner; >>> + >>> +??? /** >>> +???? * @drm_client_id: >>> +???? * >>> +???? * The client_id of the drm_file which owns the job. >>> +???? */ >>> +??? uint64_t??????????? drm_client_id; >>> ??}; >>> ??struct drm_sched_fence *to_drm_sched_fence(struct dma_fence *f); >>> @@ -629,7 +636,8 @@ drm_sched_pick_best(struct drm_gpu_scheduler >>> **sched_list, >>> ??int drm_sched_job_init(struct drm_sched_job *job, >>> ????????? ?????? struct drm_sched_entity *entity, >>> -??????? ?????? u32 credits, void *owner); >>> +??????? ?????? u32 credits, void *owner, >>> +??????? ?????? u64 drm_client_id); >>> ??void drm_sched_job_arm(struct drm_sched_job *job); >>> ??void drm_sched_entity_push_job(struct drm_sched_job *sched_job); >>> ??int drm_sched_job_add_dependency(struct drm_sched_job *job, From phasta at mailbox.org Mon May 19 11:59:01 2025 From: phasta at mailbox.org (Philipp Stanner) Date: Mon, 19 May 2025 13:59:01 +0200 Subject: [PATCH v9 02/10] drm/sched: store the drm client_id in drm_sched_fence In-Reply-To: <882eeee1-cc36-4287-8437-b39c6c84562e@damsy.net> References: <20250424083834.15518-1-pierre-eric.pelloux-prayer@amd.com> <20250424083834.15518-3-pierre-eric.pelloux-prayer@amd.com> <2d206814-968d-47df-b670-cbd8db68c5c6@damsy.net> <882eeee1-cc36-4287-8437-b39c6c84562e@damsy.net> Message-ID: <2ca3f979b0257792cfcde07fefb3ccfe60db5657.camel@mailbox.org> On Mon, 2025-05-19 at 13:02 +0200, Pierre-Eric Pelloux-Prayer wrote: > > > Le 15/05/2025 ? 08:53, Pierre-Eric Pelloux-Prayer a ?crit?: > > Hi, > > > > Le 14/05/2025 ? 14:44, Philipp Stanner a ?crit?: > > > On Thu, 2025-04-24 at 10:38 +0200, Pierre-Eric Pelloux-Prayer > > > wrote: > > > > This will be used in a later commit to trace the drm client_id > > > > in > > > > some of the gpu_scheduler trace events. > > > > > > > > This requires changing all the users of drm_sched_job_init to > > > > add an extra parameter. > > > > > > > > The newly added drm_client_id field in the drm_sched_fence is a > > > > bit > > > > of a duplicate of the owner one. One suggestion I received was > > > > to > > > > merge those 2 fields - this can't be done right now as amdgpu > > > > uses > > > > some special values (AMDGPU_FENCE_OWNER_*) that can't really be > > > > translated into a client id. Christian is working on getting > > > > rid of > > > > those; when it's done we should be able to squash > > > > owner/drm_client_id > > > > together. > > > > > > > > Reviewed-by: Christian K?nig > > > > Signed-off-by: Pierre-Eric Pelloux-Prayer > > > > > > > > --- > > > > ??drivers/accel/amdxdna/aie2_ctx.c???????????????? |? 3 ++- > > > > ??drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c?????? |? 2 +- > > > > ??drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c?????????? |? 3 ++- > > > > ??drivers/gpu/drm/amd/amdgpu/amdgpu_job.c????????? |? 8 +++++-- > > > > - > > > > ??drivers/gpu/drm/amd/amdgpu/amdgpu_job.h????????? |? 3 ++- > > > > ??drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c???? |? 2 +- > > > > ??drivers/gpu/drm/imagination/pvr_job.c??????????? |? 2 +- > > > > ??drivers/gpu/drm/imagination/pvr_queue.c????????? |? 5 +++-- > > > > ??drivers/gpu/drm/imagination/paac???????????? |? 8 +++++--- > > > > ??drivers/gpu/drm/nouveau/nouveau_sched.c????????? |? 3 ++- > > > > ??drivers/gpu/drm/panfrost/panfrost_drv.c????????? |? 2 +- > > > > ??drivers/gpu/drm/panthor/panthor_drv.c??????????? |? 3 ++- > > > > ??drivers/gpu/drm/panthor/panthor_mmu.c??????????? |? 2 +- > > > > ??drivers/gpu/drm/panthor/panthor_sched.c????????? |? 5 +++-- > > > > ??drivers/gpu/drm/panthor/panthor_sched.h????????? |? 3 ++- > > > > ??drivers/gpu/drm/scheduler/sched_fence.c????????? |? 4 ++ > > > > ??drivers/gpu/drm/scheduler/sched_internal.h?????? |? 2 +- > > > > ??drivers/gpu/drm/scheduler/sched_main.c?????????? |? 6 ++++-- > > > > ??drivers/gpu/drm/scheduler/tests/mock_scheduler.c |? 2 +- > > > > ??drivers/gpu/drm/v3d/v3d_submit.c???????????????? |? 2 +- > > > > ??drivers/gpu/drm/xe/xe_sched_job.c??????????????? |? 3 ++- > > > > ??include/drm/gpu_scheduler.h????????????????????? | 10 > > > > +++++++++- > > > > ??26 files changed, 62 insertions(+), 34 deletions(-) > > > > > > I think last time I asked about what your merge plan for this is, > > > since > > > it touches so many drivers. Should I take that? > > > > Based on: > > > > https://drm.pages.freedesktop.org/maintainer-tools/committer/committer-drm-misc.html > > > > "drm-misc is for drm core (non-driver) patches, subsystem-wide > > refactorings, > > and small trivial patches all over (including drivers)." > > > > I assume it should go through drm-misc. > > I've addressed your comments and pushed an updated branch to > https://gitlab.freedesktop.org/pepp/linux/-/commits/improve_gpu_scheduler_trace_v10 > > Any chance to get this merged soon? I took a look. Looks good! Be so kind and provide that branch as a v10 so we can conform to the process. Then I can take them, do some basic smoke tests tomorrow and then we're good to go. That should now all be good, you've got the reviews and it has settled. Thx P. > > Thanks, > Pierre-Eric > > > > > > > > > > > > > Besides one comment below, scheduler bits look fine. > > > > > > > > > > > diff --git a/drivers/accel/amdxdna/aie2_ctx.c > > > > b/drivers/accel/amdxdna/aie2_ctx.c > > > > index e04549f64d69..3e38a5f637ea 100644 > > > > --- a/drivers/accel/amdxdna/aie2_ctx.c > > > > +++ b/drivers/accel/amdxdna/aie2_ctx.c > > > > @@ -848,7 +848,8 @@ int aie2_cmd_submit(struct amdxdna_hwctx > > > > *hwctx, > > > > struct amdxdna_sched_job *job, > > > > ????????? goto up_sem; > > > > ????? } > > > > -??? ret = drm_sched_job_init(&job->base, &hwctx->priv->entity, > > > > 1, hwctx); > > > > +??? ret = drm_sched_job_init(&job->base, &hwctx->priv->entity, > > > > 1, hwctx, > > > > +???????????????? hwctx->client->filp->client_id); > > > > ????? if (ret) { > > > > ????????? XDNA_ERR(xdna, "DRM job init failed, ret %d", ret); > > > > ????????? goto free_chain; > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c > > > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c > > > > index 4cec3a873995..1a77ba7036c9 100644 > > > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c > > > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c > > > > @@ -639,7 +639,7 @@ int amdgpu_amdkfd_submit_ib(struct > > > > amdgpu_device > > > > *adev, > > > > ????????? goto err; > > > > ????? } > > > > -??? ret = amdgpu_job_alloc(adev, NULL, NULL, NULL, 1, &job); > > > > +??? ret = amdgpu_job_alloc(adev, NULL, NULL, NULL, 1, &job, > > > > 0); > > > > ????? if (ret) > > > > ????????? goto err; > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c > > > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c > > > > index 82df06a72ee0..5a231b997d65 100644 > > > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c > > > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c > > > > @@ -293,7 +293,8 @@ static int amdgpu_cs_pass1(struct > > > > amdgpu_cs_parser *p, > > > > ????? for (i = 0; i < p->gang_size; ++i) { > > > > ????????? ret = amdgpu_job_alloc(p->adev, vm, p->entities[i], > > > > vm, > > > > -??????????????? ?????? num_ibs[i], &p->jobs[i]); > > > > +??????????????? ?????? num_ibs[i], &p->jobs[i], > > > > +??????????????? ?????? p->filp->client_id); > > > > ????????? if (ret) > > > > ????????????? goto free_all_kdata; > > > > ????????? p->jobs[i]->enforce_isolation = p->adev- > > > > > enforce_isolation[fpriv->xcp_id]; > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c > > > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c > > > > index acb21fc8b3ce..75262ce8db27 100644 > > > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c > > > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c > > > > @@ -204,7 +204,8 @@ static enum drm_gpu_sched_stat > > > > amdgpu_job_timedout(struct drm_sched_job *s_job) > > > > ??int amdgpu_job_alloc(struct amdgpu_device *adev, struct > > > > amdgpu_vm > > > > *vm, > > > > ????????? ???? struct drm_sched_entity *entity, void *owner, > > > > -??????? ???? unsigned int num_ibs, struct amdgpu_job **job) > > > > +??????? ???? unsigned int num_ibs, struct amdgpu_job **job, > > > > +??????? ???? u64 drm_client_id) > > > > ??{ > > > > ????? if (num_ibs == 0) > > > > ????????? return -EINVAL; > > > > @@ -222,7 +223,8 @@ int amdgpu_job_alloc(struct amdgpu_device > > > > *adev, > > > > struct amdgpu_vm *vm, > > > > ????? if (!entity) > > > > ????????? return 0; > > > > -??? return drm_sched_job_init(&(*job)->base, entity, 1, > > > > owner); > > > > +??? return drm_sched_job_init(&(*job)->base, entity, 1, owner, > > > > +??????????????? ? drm_client_id); > > > > ??} > > > > ??int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, > > > > @@ -232,7 +234,7 @@ int amdgpu_job_alloc_with_ib(struct > > > > amdgpu_device > > > > *adev, > > > > ??{ > > > > ????? int r; > > > > -??? r = amdgpu_job_alloc(adev, NULL, entity, owner, 1, job); > > > > +??? r = amdgpu_job_alloc(adev, NULL, entity, owner, 1, job, > > > > 0); > > > > ????? if (r) > > > > ????????? return r; > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h > > > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h > > > > index ce6b9ba967ff..5a8bc6342222 100644 > > > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h > > > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h > > > > @@ -90,7 +90,8 @@ static inline struct amdgpu_ring > > > > *amdgpu_job_ring(struct amdgpu_job *job) > > > > ??int amdgpu_job_alloc(struct amdgpu_device *adev, struct > > > > amdgpu_vm > > > > *vm, > > > > ????????? ???? struct drm_sched_entity *entity, void *owner, > > > > -??????? ???? unsigned int num_ibs, struct amdgpu_job **job); > > > > +??????? ???? unsigned int num_ibs, struct amdgpu_job **job, > > > > +??????? ???? u64 drm_client_id); > > > > ??int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, > > > > ????????????? ???? struct drm_sched_entity *entity, void > > > > *owner, > > > > ????????????? ???? size_t size, enum amdgpu_ib_pool_type > > > > pool_type, > > > > diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c > > > > b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c > > > > index 3c0a5c3e0e3d..76c742328edb 100644 > > > > --- a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c > > > > +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c > > > > @@ -534,7 +534,7 @@ int etnaviv_ioctl_gem_submit(struct > > > > drm_device > > > > *dev, void *data, > > > > ????? ret = drm_sched_job_init(&submit->sched_job, > > > > ?????????????????? &ctx->sched_entity[args->pipe], > > > > -???????????????? 1, submit->ctx); > > > > +???????????????? 1, submit->ctx, file->client_id); > > > > ????? if (ret) > > > > ????????? goto err_submit_put; > > > > diff --git a/drivers/gpu/drm/imagination/pvr_job.c > > > > b/drivers/gpu/drm/imagination/pvr_job.c > > > > index 59b334d094fa..7564b0f21b42 100644 > > > > --- a/drivers/gpu/drm/imagination/pvr_job.c > > > > +++ b/drivers/gpu/drm/imagination/pvr_job.c > > > > @@ -446,7 +446,7 @@ create_job(struct pvr_device *pvr_dev, > > > > ????? if (err) > > > > ????????? goto err_put_job; > > > > -??? err = pvr_queue_job_init(job); > > > > +??? err = pvr_queue_job_init(job, pvr_file->file->client_id); > > > > ????? if (err) > > > > ????????? goto err_put_job; > > > > diff --git a/drivers/gpu/drm/imagination/pvr_queue.c > > > > b/drivers/gpu/drm/imagination/pvr_queue.c > > > > index 5e9bc0992824..5a41ee79fed6 100644 > > > > --- a/drivers/gpu/drm/imagination/pvr_queue.c > > > > +++ b/drivers/gpu/drm/imagination/pvr_queue.c > > > > @@ -1073,6 +1073,7 @@ static int > > > > pvr_queue_cleanup_fw_context(struct > > > > pvr_queue *queue) > > > > ??/** > > > > ?? * pvr_queue_job_init() - Initialize queue related fields in > > > > a > > > > pvr_job object. > > > > ?? * @job: The job to initialize. > > > > + * @drm_client_id: drm_file.client_id submitting the job > > > > ?? * > > > > ?? * Bind the job to a queue and allocate memory to guarantee > > > > pvr_queue_job_arm() > > > > ?? * and pvr_queue_job_push() can't fail. We also make sure the > > > > context type is > > > > @@ -1082,7 +1083,7 @@ static int > > > > pvr_queue_cleanup_fw_context(struct > > > > pvr_queue *queue) > > > > ?? *? * 0 on success, or > > > > ?? *? * An error code if something failed. > > > > ?? */ > > > > -int pvr_queue_job_init(struct pvr_job *job) > > > > +int pvr_queue_job_init(struct pvr_job *job, u64 drm_client_id) > > > > ??{ > > > > ????? /* Fragment jobs need at least one native fence wait on > > > > the > > > > geometry job fence. */ > > > > ????? u32 min_native_dep_count = job->type == > > > > DRM_PVR_JOB_TYPE_FRAGMENT ? 1 : 0; > > > > @@ -1099,7 +1100,7 @@ int pvr_queue_job_init(struct pvr_job > > > > *job) > > > > ????? if (!pvr_cccb_cmdseq_can_fit(&queue->cccb, > > > > job_cmds_size(job, min_native_dep_count))) > > > > ????????? return -E2BIG; > > > > -??? err = drm_sched_job_init(&job->base, &queue->entity, 1, > > > > THIS_MODULE); > > > > +??? err = drm_sched_job_init(&job->base, &queue->entity, 1, > > > > THIS_MODULE, drm_client_id); > > > > ????? if (err) > > > > ????????? return err; > > > > diff --git a/drivers/gpu/drm/imagination/pvr_queue.h > > > > b/drivers/gpu/drm/imagination/pvr_queue.h > > > > index 93fe9ac9f58c..fc1986d73fc8 100644 > > > > --- a/drivers/gpu/drm/imagination/pvr_queue.h > > > > +++ b/drivers/gpu/drm/imagination/pvr_queue.h > > > > @@ -143,7 +143,7 @@ struct pvr_queue { > > > > ??bool pvr_queue_fence_is_ufo_backed(struct dma_fence *f); > > > > -int pvr_queue_job_init(struct pvr_job *job); > > > > +int pvr_queue_job_init(struct pvr_job *job, u64 > > > > drm_client_id); > > > > ??void pvr_queue_job_cleanup(struct pvr_job *job); > > > > diff --git a/drivers/gpu/drm/lima/lima_gem.c > > > > b/drivers/gpu/drm/lima/lima_gem.c > > > > index 5deec673c11e..9722b847a539 100644 > > > > --- a/drivers/gpu/drm/lima/lima_gem.c > > > > +++ b/drivers/gpu/drm/lima/lima_gem.c > > > > @@ -341,7 +341,7 @@ int lima_gem_submit(struct drm_file *file, > > > > struct > > > > lima_submit *submit) > > > > ????? err = lima_sched_task_init( > > > > ????????? submit->task, submit->ctx->context + submit->pipe, > > > > -??????? bos, submit->nr_bos, vm); > > > > +??????? bos, submit->nr_bos, vm, file->client_id); > > > > ????? if (err) > > > > ????????? goto err_out1; > > > > diff --git a/drivers/gpu/drm/lima/lima_sched.c > > > > b/drivers/gpu/drm/lima/lima_sched.c > > > > index 7934098e651b..954f4325b859 100644 > > > > --- a/drivers/gpu/drm/lima/lima_sched.c > > > > +++ b/drivers/gpu/drm/lima/lima_sched.c > > > > @@ -113,7 +113,8 @@ static inline struct lima_sched_pipe > > > > *to_lima_pipe(struct drm_gpu_scheduler *sch > > > > ??int lima_sched_task_init(struct lima_sched_task *task, > > > > ?????????????? struct lima_sched_context *context, > > > > ?????????????? struct lima_bo **bos, int num_bos, > > > > -???????????? struct lima_vm *vm) > > > > +???????????? struct lima_vm *vm, > > > > +???????????? u64 drm_client_id) > > > > ??{ > > > > ????? int err, i; > > > > @@ -124,7 +125,8 @@ int lima_sched_task_init(struct > > > > lima_sched_task > > > > *task, > > > > ????? for (i = 0; i < num_bos; i++) > > > > ????????? drm_gem_object_get(&bos[i]->base.base); > > > > -??? err = drm_sched_job_init(&task->base, &context->base, 1, > > > > vm); > > > > +??? err = drm_sched_job_init(&task->base, &context->base, 1, > > > > vm, > > > > +???????????????? drm_client_id); > > > > ????? if (err) { > > > > ????????? kfree(task->bos); > > > > ????????? return err; > > > > diff --git a/drivers/gpu/drm/lima/lima_sched.h > > > > b/drivers/gpu/drm/lima/lima_sched.h > > > > index 85b23ba901d5..1a08faf8a529 100644 > > > > --- a/drivers/gpu/drm/lima/lima_sched.h > > > > +++ b/drivers/gpu/drm/lima/lima_sched.h > > > > @@ -88,7 +88,8 @@ struct lima_sched_pipe { > > > > ??int lima_sched_task_init(struct lima_sched_task *task, > > > > ?????????????? struct lima_sched_context *context, > > > > ?????????????? struct lima_bo **bos, int num_bos, > > > > -???????????? struct lima_vm *vm); > > > > +???????????? struct lima_vm *vm, > > > > +???????????? u64 drm_client_id); > > > > ??void lima_sched_task_fini(struct lima_sched_task *task); > > > > ??int lima_sched_context_init(struct lima_sched_pipe *pipe, > > > > diff --git a/drivers/gpu/drm/msm/msm_gem_submit.c > > > > b/drivers/gpu/drm/msm/msm_gem_submit.c > > > > index 3e9aa2cc38ef..d9be0fe3d674 100644 > > > > --- a/drivers/gpu/drm/msm/msm_gem_submit.c > > > > +++ b/drivers/gpu/drm/msm/msm_gem_submit.c > > > > @@ -30,7 +30,7 @@ > > > > ??static struct msm_gem_submit *submit_create(struct drm_device > > > > *dev, > > > > ????????? struct msm_gpu *gpu, > > > > ????????? struct msm_gpu_submitqueue *queue, uint32_t nr_bos, > > > > -??????? uint32_t nr_cmds) > > > > +??????? uint32_t nr_cmds, u64 drm_client_id) > > > > ??{ > > > > ????? static atomic_t ident = ATOMIC_INIT(0); > > > > ????? struct msm_gem_submit *submit; > > > > @@ -54,7 +54,8 @@ static struct msm_gem_submit > > > > *submit_create(struct > > > > drm_device *dev, > > > > ????????? return ERR_PTR(ret); > > > > ????? } > > > > -??? ret = drm_sched_job_init(&submit->base, queue->entity, 1, > > > > queue); > > > > +??? ret = drm_sched_job_init(&submit->base, queue->entity, 1, > > > > queue, > > > > +???????????????? drm_client_id); > > > > ????? if (ret) { > > > > ????????? kfree(submit->hw_fence); > > > > ????????? kfree(submit); > > > > @@ -693,7 +694,8 @@ int msm_ioctl_gem_submit(struct drm_device > > > > *dev, > > > > void *data, > > > > ????????? } > > > > ????? } > > > > -??? submit = submit_create(dev, gpu, queue, args->nr_bos, > > > > args- > > > > > nr_cmds); > > > > +??? submit = submit_create(dev, gpu, queue, args->nr_bos, > > > > args- > > > > > nr_cmds, > > > > +??????????? ?????? file->client_id); > > > > ????? if (IS_ERR(submit)) { > > > > ????????? ret = PTR_ERR(submit); > > > > ????????? goto out_post_unlock; > > > > diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.c > > > > b/drivers/gpu/drm/nouveau/nouveau_sched.c > > > > index d326e55d2d24..460a5fb02412 100644 > > > > --- a/drivers/gpu/drm/nouveau/nouveau_sched.c > > > > +++ b/drivers/gpu/drm/nouveau/nouveau_sched.c > > > > @@ -87,7 +87,8 @@ nouveau_job_init(struct nouveau_job *job, > > > > ????? } > > > > ????? ret = drm_sched_job_init(&job->base, &sched->entity, > > > > -???????????????? args->credits, NULL); > > > > +???????????????? args->credits, NULL, > > > > +???????????????? job->file_priv->client_id); > > > > ????? if (ret) > > > > ????????? goto err_free_chains; > > > > diff --git a/drivers/gpu/drm/panfrost/panfrost_drv.c > > > > b/drivers/gpu/drm/panfrost/panfrost_drv.c > > > > index b87f83e94eda..d5c2c6530ed8 100644 > > > > --- a/drivers/gpu/drm/panfrost/panfrost_drv.c > > > > +++ b/drivers/gpu/drm/panfrost/panfrost_drv.c > > > > @@ -312,7 +312,7 @@ static int panfrost_ioctl_submit(struct > > > > drm_device *dev, void *data, > > > > ????? ret = drm_sched_job_init(&job->base, > > > > ?????????????????? &file_priv->sched_entity[slot], > > > > -???????????????? 1, NULL); > > > > +???????????????? 1, NULL, file->client_id); > > > > ????? if (ret) > > > > ????????? goto out_put_job; > > > > diff --git a/drivers/gpu/drm/panthor/panthor_drv.c > > > > b/drivers/gpu/drm/panthor/panthor_drv.c > > > > index 06fe46e32073..bd8e1900c919 100644 > > > > --- a/drivers/gpu/drm/panthor/panthor_drv.c > > > > +++ b/drivers/gpu/drm/panthor/panthor_drv.c > > > > @@ -989,7 +989,8 @@ static int > > > > panthor_ioctl_group_submit(struct > > > > drm_device *ddev, void *data, > > > > ????????? const struct drm_panthor_queue_submit *qsubmit = > > > > &jobs_args[i]; > > > > ????????? struct drm_sched_job *job; > > > > -??????? job = panthor_job_create(pfile, args->group_handle, > > > > qsubmit); > > > > +??????? job = panthor_job_create(pfile, args->group_handle, > > > > qsubmit, > > > > +???????????????????? file->client_id); > > > > ????????? if (IS_ERR(job)) { > > > > ????????????? ret = PTR_ERR(job); > > > > ????????????? goto out_cleanup_submit_ctx; > > > > diff --git a/drivers/gpu/drm/panthor/panthor_mmu.c > > > > b/drivers/gpu/drm/panthor/panthor_mmu.c > > > > index 12a02e28f50f..e0c79bd2d173 100644 > > > > --- a/drivers/gpu/drm/panthor/panthor_mmu.c > > > > +++ b/drivers/gpu/drm/panthor/panthor_mmu.c > > > > @@ -2516,7 +2516,7 @@ panthor_vm_bind_job_create(struct > > > > drm_file > > > > *file, > > > > ????? kref_init(&job->refcount); > > > > ????? job->vm = panthor_vm_get(vm); > > > > -??? ret = drm_sched_job_init(&job->base, &vm->entity, 1, vm); > > > > +??? ret = drm_sched_job_init(&job->base, &vm->entity, 1, vm, > > > > file->client_id); > > > > ????? if (ret) > > > > ????????? goto err_put_job; > > > > diff --git a/drivers/gpu/drm/panthor/panthor_sched.c > > > > b/drivers/gpu/drm/panthor/panthor_sched.c > > > > index 446ec780eb4a..2af860c9068a 100644 > > > > --- a/drivers/gpu/drm/panthor/panthor_sched.c > > > > +++ b/drivers/gpu/drm/panthor/panthor_sched.c > > > > @@ -3729,7 +3729,8 @@ struct panthor_vm *panthor_job_vm(struct > > > > drm_sched_job *sched_job) > > > > ??struct drm_sched_job * > > > > ??panthor_job_create(struct panthor_file *pfile, > > > > ????????? ?? u16 group_handle, > > > > -??????? ?? const struct drm_panthor_queue_submit *qsubmit) > > > > +??????? ?? const struct drm_panthor_queue_submit *qsubmit, > > > > +??????? ?? u64 drm_client_id) > > > > ??{ > > > > ????? struct panthor_group_pool *gpool = pfile->groups; > > > > ????? struct panthor_job *job; > > > > @@ -3801,7 +3802,7 @@ panthor_job_create(struct panthor_file > > > > *pfile, > > > > ????? ret = drm_sched_job_init(&job->base, > > > > ?????????????????? &job->group->queues[job- > > > > > queue_idx]->entity, > > > > -???????????????? credits, job->group); > > > > +???????????????? credits, job->group, > > > > drm_client_id); > > > > ????? if (ret) > > > > ????????? goto err_put_job; > > > > diff --git a/drivers/gpu/drm/panthor/panthor_sched.h > > > > b/drivers/gpu/drm/panthor/panthor_sched.h > > > > index e650a445cf50..742b0b4ff3a3 100644 > > > > --- a/drivers/gpu/drm/panthor/panthor_sched.h > > > > +++ b/drivers/gpu/drm/panthor/panthor_sched.h > > > > @@ -29,7 +29,8 @@ int panthor_group_get_state(struct > > > > panthor_file > > > > *pfile, > > > > ??struct drm_sched_job * > > > > ??panthor_job_create(struct panthor_file *pfile, > > > > ????????? ?? u16 group_handle, > > > > -??????? ?? const struct drm_panthor_queue_submit *qsubmit); > > > > +??????? ?? const struct drm_panthor_queue_submit *qsubmit, > > > > +??????? ?? u64 drm_client_id); > > > > ??struct drm_sched_job *panthor_job_get(struct drm_sched_job > > > > *job); > > > > ??struct panthor_vm *panthor_job_vm(struct drm_sched_job > > > > *sched_job); > > > > ??void panthor_job_put(struct drm_sched_job *job); > > > > diff --git a/drivers/gpu/drm/scheduler/sched_fence.c > > > > b/drivers/gpu/drm/scheduler/sched_fence.c > > > > index e971528504a5..d208d384d38d 100644 > > > > --- a/drivers/gpu/drm/scheduler/sched_fence.c > > > > +++ b/drivers/gpu/drm/scheduler/sched_fence.c > > > > @@ -206,7 +206,8 @@ struct drm_sched_fence > > > > *to_drm_sched_fence(struct > > > > dma_fence *f) > > > > ??EXPORT_SYMBOL(to_drm_sched_fence); > > > > ??struct drm_sched_fence *drm_sched_fence_alloc(struct > > > > drm_sched_entity *entity, > > > > -??????????????????? ????? void *owner) > > > > +??????????????????? ????? void *owner, > > > > +??????????????????? ????? u64 drm_client_id) > > > > ??{ > > > > ????? struct drm_sched_fence *fence = NULL; > > > > @@ -215,6 +216,7 @@ struct drm_sched_fence > > > > *drm_sched_fence_alloc(struct drm_sched_entity *entity, > > > > ????????? return NULL; > > > > ????? fence->owner = owner; > > > > +??? fence->drm_client_id = drm_client_id; > > > > ????? spin_lock_init(&fence->lock); > > > > ????? return fence; > > > > diff --git a/drivers/gpu/drm/scheduler/sched_internal.h > > > > b/drivers/gpu/drm/scheduler/sched_internal.h > > > > index 599cf6e1bb74..7ea5a6736f98 100644 > > > > --- a/drivers/gpu/drm/scheduler/sched_internal.h > > > > +++ b/drivers/gpu/drm/scheduler/sched_internal.h > > > > @@ -24,7 +24,7 @@ void drm_sched_entity_select_rq(struct > > > > drm_sched_entity *entity); > > > > ??struct drm_sched_job *drm_sched_entity_pop_job(struct > > > > drm_sched_entity *entity); > > > > ??struct drm_sched_fence *drm_sched_fence_alloc(struct > > > > drm_sched_entity *s_entity, > > > > -??????????????????? ????? void *owner); > > > > +??????????????????? ????? void *owner, u64 > > > > drm_client_id); > > > > ??void drm_sched_fence_init(struct drm_sched_fence *fence, > > > > ????????????? ? struct drm_sched_entity *entity); > > > > ??void drm_sched_fence_free(struct drm_sched_fence *fence); > > > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c > > > > b/drivers/gpu/drm/scheduler/sched_main.c > > > > index 829579c41c6b..60611618f3ab 100644 > > > > --- a/drivers/gpu/drm/scheduler/sched_main.c > > > > +++ b/drivers/gpu/drm/scheduler/sched_main.c > > > > @@ -764,6 +764,7 @@ EXPORT_SYMBOL(drm_sched_resubmit_jobs); > > > > ?? * @credits: the number of credits this job contributes to > > > > the > > > > schedulers > > > > ?? * credit limit > > > > ?? * @owner: job owner for debugging > > > > + * @drm_client_id: drm_file.client_id of the owner > > > > > > For the docu generation to link that properly it must be written > > > as > > > > > > &struct drm_file.client_id > > > > Noted. > > > > > > > > Besides, if this were an optional parameter, one should document > > > it. > > > I'm not sure if it is, I haven't used these client_id's before. > > > > Passing an invalid client_id would only cause the trace events to > > print the invalid client_id. > > > > Thanks, > > Pierre-Eric > > > > > > > > > > P. > > > > > > > ?? * > > > > ?? * Refer to drm_sched_entity_push_job() documentation > > > > ?? * for locking considerations. > > > > @@ -784,7 +785,8 @@ EXPORT_SYMBOL(drm_sched_resubmit_jobs); > > > > ?? */ > > > > ??int drm_sched_job_init(struct drm_sched_job *job, > > > > ????????? ?????? struct drm_sched_entity *entity, > > > > -??????? ?????? u32 credits, void *owner) > > > > +??????? ?????? u32 credits, void *owner, > > > > +??????? ?????? uint64_t drm_client_id) > > > > ??{ > > > > ????? if (!entity->rq) { > > > > ????????? /* This will most likely be followed by missing > > > > frames > > > > @@ -810,7 +812,7 @@ int drm_sched_job_init(struct drm_sched_job > > > > *job, > > > > ????? job->entity = entity; > > > > ????? job->credits = credits; > > > > -??? job->s_fence = drm_sched_fence_alloc(entity, owner); > > > > +??? job->s_fence = drm_sched_fence_alloc(entity, owner, > > > > drm_client_id); > > > > ????? if (!job->s_fence) > > > > ????????? return -ENOMEM; > > > > diff --git a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c > > > > b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c > > > > index f999c8859cf7..09ffbdb32d76 100644 > > > > --- a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c > > > > +++ b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c > > > > @@ -35,7 +35,7 @@ drm_mock_sched_entity_new(struct kunit *test, > > > > ????? ret = drm_sched_entity_init(&entity->base, > > > > ????????????????? ??? priority, > > > > ????????????????? ??? &drm_sched, 1, > > > > -??????????????? ??? NULL); > > > > +??????????????? ??? NULL, 1); > > > > ????? KUNIT_ASSERT_EQ(test, ret, 0); > > > > ????? entity->test = test; > > > > diff --git a/drivers/gpu/drm/v3d/v3d_submit.c > > > > b/drivers/gpu/drm/v3d/v3d_submit.c > > > > index 4ff5de46fb22..5171ffe9012d 100644 > > > > --- a/drivers/gpu/drm/v3d/v3d_submit.c > > > > +++ b/drivers/gpu/drm/v3d/v3d_submit.c > > > > @@ -169,7 +169,7 @@ v3d_job_init(struct v3d_dev *v3d, struct > > > > drm_file > > > > *file_priv, > > > > ????? job->file = file_priv; > > > > ????? ret = drm_sched_job_init(&job->base, &v3d_priv- > > > > > sched_entity[queue], > > > > -???????????????? 1, v3d_priv); > > > > +???????????????? 1, v3d_priv, file_priv->client_id); > > > > ????? if (ret) > > > > ????????? return ret; > > > > diff --git a/drivers/gpu/drm/xe/xe_sched_job.c > > > > b/drivers/gpu/drm/xe/xe_sched_job.c > > > > index 1905ca590965..f4679cb9a56b 100644 > > > > --- a/drivers/gpu/drm/xe/xe_sched_job.c > > > > +++ b/drivers/gpu/drm/xe/xe_sched_job.c > > > > @@ -113,7 +113,8 @@ struct xe_sched_job > > > > *xe_sched_job_create(struct > > > > xe_exec_queue *q, > > > > ????? kref_init(&job->refcount); > > > > ????? xe_exec_queue_get(job->q); > > > > -??? err = drm_sched_job_init(&job->drm, q->entity, 1, NULL); > > > > +??? err = drm_sched_job_init(&job->drm, q->entity, 1, NULL, > > > > +???????????????? q->xef->drm->client_id); > > > > ????? if (err) > > > > ????????? goto err_free; > > > > diff --git a/include/drm/gpu_scheduler.h > > > > b/include/drm/gpu_scheduler.h > > > > index 1a7e377d4cbb..6fe3b4c0cffb 100644 > > > > --- a/include/drm/gpu_scheduler.h > > > > +++ b/include/drm/gpu_scheduler.h > > > > @@ -305,6 +305,13 @@ struct drm_sched_fence { > > > > ?????????? * @owner: job owner for debugging > > > > ?????????? */ > > > > ????? void??????????????? *owner; > > > > + > > > > +??? /** > > > > +???? * @drm_client_id: > > > > +???? * > > > > +???? * The client_id of the drm_file which owns the job. > > > > +???? */ > > > > +??? uint64_t??????????? drm_client_id; > > > > ??}; > > > > ??struct drm_sched_fence *to_drm_sched_fence(struct dma_fence > > > > *f); > > > > @@ -629,7 +636,8 @@ drm_sched_pick_best(struct > > > > drm_gpu_scheduler > > > > **sched_list, > > > > ??int drm_sched_job_init(struct drm_sched_job *job, > > > > ????????? ?????? struct drm_sched_entity *entity, > > > > -??????? ?????? u32 credits, void *owner); > > > > +??????? ?????? u32 credits, void *owner, > > > > +??????? ?????? u64 drm_client_id); > > > > ??void drm_sched_job_arm(struct drm_sched_job *job); > > > > ??void drm_sched_entity_push_job(struct drm_sched_job > > > > *sched_job); > > > > ??int drm_sched_job_add_dependency(struct drm_sched_job *job, From pierre-eric.pelloux-prayer at amd.com Wed May 21 15:45:02 2025 From: pierre-eric.pelloux-prayer at amd.com (Pierre-Eric Pelloux-Prayer) Date: Wed, 21 May 2025 17:45:02 +0200 Subject: [PATCH v10 00/10] Improve gpu_scheduler trace events + UAPI Message-ID: <20250521154531.10541-1-pierre-eric.pelloux-prayer@amd.com> Hi, The initial goal of this series was to improve the drm and amdgpu trace events to be able to expose more of the inner workings of the scheduler and drivers to developers via tools. Then, the series evolved to become focused only on gpu_scheduler. The changes around vblank events will be part of a different series, as well as the amdgpu ones. Moreover Sima suggested to make some trace events stable uAPI, so tools can rely on them long term. The first patches extend and cleanup the gpu scheduler events, then add a documentation entry in drm-uapi.rst. The last 2 patches are new in v8. One is based on a suggestion from Tvrtko and gets rid of drm_sched_job::id. The other is a cleanup of amdgpu trace events to use the fence=%llu:%llu format. The drm_sched_job patches don't affect gpuvis which has code to parse the gpu_scheduler events but these events are not enabled. Changes since v9: * fixed documentation link syntax * fixed typos in commit messages * spelled out that these events cannot be used before drm_sched_job_arm has been called Changes since v8: * swapped patches 8 & 9 * rebased on drm-next Changes since v7: * uint64_t -> u64 * reworked dependencies tracing (Tvrtko) * use common name prefix for all events (Tvrtko) * dropped drm_sched_job::id (Tvrtko) Useful links: - userspace tool using the updated events: https://gitlab.freedesktop.org/tomstdenis/umr/-/merge_requests/37 - v8: https://lists.freedesktop.org/archives/dri-devel/2025-March/496781.html Pierre-Eric Pelloux-Prayer (10): drm/debugfs: Output client_id in in drm_clients_info drm/sched: Store the drm client_id in drm_sched_fence drm/sched: Add device name to the drm_sched_process_job event drm/sched: Cleanup gpu_scheduler trace events drm/sched: Trace dependencies for GPU jobs drm/sched: Add the drm_client_id to the drm_sched_run/exec_job events drm/sched: Cleanup event names drm: Get rid of drm_sched_job.id drm/doc: Document some tracepoints as uAPI drm/amdgpu: update trace format to match gpu_scheduler_trace Documentation/gpu/drm-uapi.rst | 19 ++++ drivers/accel/amdxdna/aie2_ctx.c | 3 +- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 3 +- drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 8 +- drivers/gpu/drm/amd/amdgpu/amdgpu_job.h | 3 +- drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h | 32 ++---- drivers/gpu/drm/drm_debugfs.c | 10 +- drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c | 2 +- drivers/gpu/drm/imagination/pvr_job.c | 2 +- drivers/gpu/drm/imagination/pvr_queue.c | 5 +- drivers/gpu/drm/imagination/pvr_queue.h | 2 +- drivers/gpu/drm/lima/lima_gem.c | 2 +- drivers/gpu/drm/lima/lima_sched.c | 6 +- drivers/gpu/drm/lima/lima_sched.h | 3 +- drivers/gpu/drm/msm/msm_gem_submit.c | 8 +- drivers/gpu/drm/nouveau/nouveau_sched.c | 3 +- drivers/gpu/drm/panfrost/panfrost_drv.c | 2 +- drivers/gpu/drm/panthor/panthor_drv.c | 3 +- drivers/gpu/drm/panthor/panthor_mmu.c | 2 +- drivers/gpu/drm/panthor/panthor_sched.c | 5 +- drivers/gpu/drm/panthor/panthor_sched.h | 3 +- .../gpu/drm/scheduler/gpu_scheduler_trace.h | 103 +++++++++++++----- drivers/gpu/drm/scheduler/sched_entity.c | 16 ++- drivers/gpu/drm/scheduler/sched_fence.c | 4 +- drivers/gpu/drm/scheduler/sched_internal.h | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 12 +- .../gpu/drm/scheduler/tests/mock_scheduler.c | 2 +- drivers/gpu/drm/v3d/v3d_submit.c | 2 +- drivers/gpu/drm/xe/xe_sched_job.c | 3 +- include/drm/gpu_scheduler.h | 13 ++- 31 files changed, 188 insertions(+), 97 deletions(-) -- 2.43.0 From pierre-eric.pelloux-prayer at amd.com Wed May 21 15:45:04 2025 From: pierre-eric.pelloux-prayer at amd.com (Pierre-Eric Pelloux-Prayer) Date: Wed, 21 May 2025 17:45:04 +0200 Subject: [PATCH v10 02/10] drm/sched: Store the drm client_id in drm_sched_fence In-Reply-To: <20250521154531.10541-1-pierre-eric.pelloux-prayer@amd.com> References: <20250521154531.10541-1-pierre-eric.pelloux-prayer@amd.com> Message-ID: <20250521154531.10541-3-pierre-eric.pelloux-prayer@amd.com> This will be used in a later commit to trace the drm client_id in some of the gpu_scheduler trace events. This requires changing all the users of drm_sched_job_init to add an extra parameter. The newly added drm_client_id field in the drm_sched_fence is a bit of a duplicate of the owner one. One suggestion I received was to merge those 2 fields - this can't be done right now as amdgpu uses some special values (AMDGPU_FENCE_OWNER_*) that can't really be translated into a client id. Christian is working on getting rid of those; when it's done we should be able to squash owner/drm_client_id together. Reviewed-by: Christian K?nig Signed-off-by: Pierre-Eric Pelloux-Prayer --- drivers/accel/amdxdna/aie2_ctx.c | 3 ++- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 3 ++- drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 8 +++++--- drivers/gpu/drm/amd/amdgpu/amdgpu_job.h | 3 ++- drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c | 2 +- drivers/gpu/drm/imagination/pvr_job.c | 2 +- drivers/gpu/drm/imagination/pvr_queue.c | 5 +++-- drivers/gpu/drm/imagination/pvr_queue.h | 2 +- drivers/gpu/drm/lima/lima_gem.c | 2 +- drivers/gpu/drm/lima/lima_sched.c | 6 ++++-- drivers/gpu/drm/lima/lima_sched.h | 3 ++- drivers/gpu/drm/msm/msm_gem_submit.c | 8 +++++--- drivers/gpu/drm/nouveau/nouveau_sched.c | 3 ++- drivers/gpu/drm/panfrost/panfrost_drv.c | 2 +- drivers/gpu/drm/panthor/panthor_drv.c | 3 ++- drivers/gpu/drm/panthor/panthor_mmu.c | 2 +- drivers/gpu/drm/panthor/panthor_sched.c | 5 +++-- drivers/gpu/drm/panthor/panthor_sched.h | 3 ++- drivers/gpu/drm/scheduler/sched_fence.c | 4 +++- drivers/gpu/drm/scheduler/sched_internal.h | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 7 +++++-- drivers/gpu/drm/scheduler/tests/mock_scheduler.c | 2 +- drivers/gpu/drm/v3d/v3d_submit.c | 2 +- drivers/gpu/drm/xe/xe_sched_job.c | 3 ++- include/drm/gpu_scheduler.h | 10 +++++++++- 26 files changed, 63 insertions(+), 34 deletions(-) diff --git a/drivers/accel/amdxdna/aie2_ctx.c b/drivers/accel/amdxdna/aie2_ctx.c index e04549f64d69..3e38a5f637ea 100644 --- a/drivers/accel/amdxdna/aie2_ctx.c +++ b/drivers/accel/amdxdna/aie2_ctx.c @@ -848,7 +848,8 @@ int aie2_cmd_submit(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, goto up_sem; } - ret = drm_sched_job_init(&job->base, &hwctx->priv->entity, 1, hwctx); + ret = drm_sched_job_init(&job->base, &hwctx->priv->entity, 1, hwctx, + hwctx->client->filp->client_id); if (ret) { XDNA_ERR(xdna, "DRM job init failed, ret %d", ret); goto free_chain; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c index 4cec3a873995..1a77ba7036c9 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c @@ -639,7 +639,7 @@ int amdgpu_amdkfd_submit_ib(struct amdgpu_device *adev, goto err; } - ret = amdgpu_job_alloc(adev, NULL, NULL, NULL, 1, &job); + ret = amdgpu_job_alloc(adev, NULL, NULL, NULL, 1, &job, 0); if (ret) goto err; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c index 82df06a72ee0..5a231b997d65 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c @@ -293,7 +293,8 @@ static int amdgpu_cs_pass1(struct amdgpu_cs_parser *p, for (i = 0; i < p->gang_size; ++i) { ret = amdgpu_job_alloc(p->adev, vm, p->entities[i], vm, - num_ibs[i], &p->jobs[i]); + num_ibs[i], &p->jobs[i], + p->filp->client_id); if (ret) goto free_all_kdata; p->jobs[i]->enforce_isolation = p->adev->enforce_isolation[fpriv->xcp_id]; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c index acb21fc8b3ce..75262ce8db27 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c @@ -204,7 +204,8 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job) int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm *vm, struct drm_sched_entity *entity, void *owner, - unsigned int num_ibs, struct amdgpu_job **job) + unsigned int num_ibs, struct amdgpu_job **job, + u64 drm_client_id) { if (num_ibs == 0) return -EINVAL; @@ -222,7 +223,8 @@ int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm *vm, if (!entity) return 0; - return drm_sched_job_init(&(*job)->base, entity, 1, owner); + return drm_sched_job_init(&(*job)->base, entity, 1, owner, + drm_client_id); } int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, @@ -232,7 +234,7 @@ int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, { int r; - r = amdgpu_job_alloc(adev, NULL, entity, owner, 1, job); + r = amdgpu_job_alloc(adev, NULL, entity, owner, 1, job, 0); if (r) return r; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h index ce6b9ba967ff..5a8bc6342222 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h @@ -90,7 +90,8 @@ static inline struct amdgpu_ring *amdgpu_job_ring(struct amdgpu_job *job) int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm *vm, struct drm_sched_entity *entity, void *owner, - unsigned int num_ibs, struct amdgpu_job **job); + unsigned int num_ibs, struct amdgpu_job **job, + u64 drm_client_id); int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, struct drm_sched_entity *entity, void *owner, size_t size, enum amdgpu_ib_pool_type pool_type, diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c index 3c0a5c3e0e3d..76c742328edb 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c @@ -534,7 +534,7 @@ int etnaviv_ioctl_gem_submit(struct drm_device *dev, void *data, ret = drm_sched_job_init(&submit->sched_job, &ctx->sched_entity[args->pipe], - 1, submit->ctx); + 1, submit->ctx, file->client_id); if (ret) goto err_submit_put; diff --git a/drivers/gpu/drm/imagination/pvr_job.c b/drivers/gpu/drm/imagination/pvr_job.c index 59b334d094fa..7564b0f21b42 100644 --- a/drivers/gpu/drm/imagination/pvr_job.c +++ b/drivers/gpu/drm/imagination/pvr_job.c @@ -446,7 +446,7 @@ create_job(struct pvr_device *pvr_dev, if (err) goto err_put_job; - err = pvr_queue_job_init(job); + err = pvr_queue_job_init(job, pvr_file->file->client_id); if (err) goto err_put_job; diff --git a/drivers/gpu/drm/imagination/pvr_queue.c b/drivers/gpu/drm/imagination/pvr_queue.c index 5e9bc0992824..5a41ee79fed6 100644 --- a/drivers/gpu/drm/imagination/pvr_queue.c +++ b/drivers/gpu/drm/imagination/pvr_queue.c @@ -1073,6 +1073,7 @@ static int pvr_queue_cleanup_fw_context(struct pvr_queue *queue) /** * pvr_queue_job_init() - Initialize queue related fields in a pvr_job object. * @job: The job to initialize. + * @drm_client_id: drm_file.client_id submitting the job * * Bind the job to a queue and allocate memory to guarantee pvr_queue_job_arm() * and pvr_queue_job_push() can't fail. We also make sure the context type is @@ -1082,7 +1083,7 @@ static int pvr_queue_cleanup_fw_context(struct pvr_queue *queue) * * 0 on success, or * * An error code if something failed. */ -int pvr_queue_job_init(struct pvr_job *job) +int pvr_queue_job_init(struct pvr_job *job, u64 drm_client_id) { /* Fragment jobs need at least one native fence wait on the geometry job fence. */ u32 min_native_dep_count = job->type == DRM_PVR_JOB_TYPE_FRAGMENT ? 1 : 0; @@ -1099,7 +1100,7 @@ int pvr_queue_job_init(struct pvr_job *job) if (!pvr_cccb_cmdseq_can_fit(&queue->cccb, job_cmds_size(job, min_native_dep_count))) return -E2BIG; - err = drm_sched_job_init(&job->base, &queue->entity, 1, THIS_MODULE); + err = drm_sched_job_init(&job->base, &queue->entity, 1, THIS_MODULE, drm_client_id); if (err) return err; diff --git a/drivers/gpu/drm/imagination/pvr_queue.h b/drivers/gpu/drm/imagination/pvr_queue.h index 93fe9ac9f58c..fc1986d73fc8 100644 --- a/drivers/gpu/drm/imagination/pvr_queue.h +++ b/drivers/gpu/drm/imagination/pvr_queue.h @@ -143,7 +143,7 @@ struct pvr_queue { bool pvr_queue_fence_is_ufo_backed(struct dma_fence *f); -int pvr_queue_job_init(struct pvr_job *job); +int pvr_queue_job_init(struct pvr_job *job, u64 drm_client_id); void pvr_queue_job_cleanup(struct pvr_job *job); diff --git a/drivers/gpu/drm/lima/lima_gem.c b/drivers/gpu/drm/lima/lima_gem.c index 5deec673c11e..9722b847a539 100644 --- a/drivers/gpu/drm/lima/lima_gem.c +++ b/drivers/gpu/drm/lima/lima_gem.c @@ -341,7 +341,7 @@ int lima_gem_submit(struct drm_file *file, struct lima_submit *submit) err = lima_sched_task_init( submit->task, submit->ctx->context + submit->pipe, - bos, submit->nr_bos, vm); + bos, submit->nr_bos, vm, file->client_id); if (err) goto err_out1; diff --git a/drivers/gpu/drm/lima/lima_sched.c b/drivers/gpu/drm/lima/lima_sched.c index 7934098e651b..954f4325b859 100644 --- a/drivers/gpu/drm/lima/lima_sched.c +++ b/drivers/gpu/drm/lima/lima_sched.c @@ -113,7 +113,8 @@ static inline struct lima_sched_pipe *to_lima_pipe(struct drm_gpu_scheduler *sch int lima_sched_task_init(struct lima_sched_task *task, struct lima_sched_context *context, struct lima_bo **bos, int num_bos, - struct lima_vm *vm) + struct lima_vm *vm, + u64 drm_client_id) { int err, i; @@ -124,7 +125,8 @@ int lima_sched_task_init(struct lima_sched_task *task, for (i = 0; i < num_bos; i++) drm_gem_object_get(&bos[i]->base.base); - err = drm_sched_job_init(&task->base, &context->base, 1, vm); + err = drm_sched_job_init(&task->base, &context->base, 1, vm, + drm_client_id); if (err) { kfree(task->bos); return err; diff --git a/drivers/gpu/drm/lima/lima_sched.h b/drivers/gpu/drm/lima/lima_sched.h index 85b23ba901d5..1a08faf8a529 100644 --- a/drivers/gpu/drm/lima/lima_sched.h +++ b/drivers/gpu/drm/lima/lima_sched.h @@ -88,7 +88,8 @@ struct lima_sched_pipe { int lima_sched_task_init(struct lima_sched_task *task, struct lima_sched_context *context, struct lima_bo **bos, int num_bos, - struct lima_vm *vm); + struct lima_vm *vm, + u64 drm_client_id); void lima_sched_task_fini(struct lima_sched_task *task); int lima_sched_context_init(struct lima_sched_pipe *pipe, diff --git a/drivers/gpu/drm/msm/msm_gem_submit.c b/drivers/gpu/drm/msm/msm_gem_submit.c index 3e9aa2cc38ef..d9be0fe3d674 100644 --- a/drivers/gpu/drm/msm/msm_gem_submit.c +++ b/drivers/gpu/drm/msm/msm_gem_submit.c @@ -30,7 +30,7 @@ static struct msm_gem_submit *submit_create(struct drm_device *dev, struct msm_gpu *gpu, struct msm_gpu_submitqueue *queue, uint32_t nr_bos, - uint32_t nr_cmds) + uint32_t nr_cmds, u64 drm_client_id) { static atomic_t ident = ATOMIC_INIT(0); struct msm_gem_submit *submit; @@ -54,7 +54,8 @@ static struct msm_gem_submit *submit_create(struct drm_device *dev, return ERR_PTR(ret); } - ret = drm_sched_job_init(&submit->base, queue->entity, 1, queue); + ret = drm_sched_job_init(&submit->base, queue->entity, 1, queue, + drm_client_id); if (ret) { kfree(submit->hw_fence); kfree(submit); @@ -693,7 +694,8 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data, } } - submit = submit_create(dev, gpu, queue, args->nr_bos, args->nr_cmds); + submit = submit_create(dev, gpu, queue, args->nr_bos, args->nr_cmds, + file->client_id); if (IS_ERR(submit)) { ret = PTR_ERR(submit); goto out_post_unlock; diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.c b/drivers/gpu/drm/nouveau/nouveau_sched.c index d326e55d2d24..460a5fb02412 100644 --- a/drivers/gpu/drm/nouveau/nouveau_sched.c +++ b/drivers/gpu/drm/nouveau/nouveau_sched.c @@ -87,7 +87,8 @@ nouveau_job_init(struct nouveau_job *job, } ret = drm_sched_job_init(&job->base, &sched->entity, - args->credits, NULL); + args->credits, NULL, + job->file_priv->client_id); if (ret) goto err_free_chains; diff --git a/drivers/gpu/drm/panfrost/panfrost_drv.c b/drivers/gpu/drm/panfrost/panfrost_drv.c index b87f83e94eda..d5c2c6530ed8 100644 --- a/drivers/gpu/drm/panfrost/panfrost_drv.c +++ b/drivers/gpu/drm/panfrost/panfrost_drv.c @@ -312,7 +312,7 @@ static int panfrost_ioctl_submit(struct drm_device *dev, void *data, ret = drm_sched_job_init(&job->base, &file_priv->sched_entity[slot], - 1, NULL); + 1, NULL, file->client_id); if (ret) goto out_put_job; diff --git a/drivers/gpu/drm/panthor/panthor_drv.c b/drivers/gpu/drm/panthor/panthor_drv.c index 06fe46e32073..bd8e1900c919 100644 --- a/drivers/gpu/drm/panthor/panthor_drv.c +++ b/drivers/gpu/drm/panthor/panthor_drv.c @@ -989,7 +989,8 @@ static int panthor_ioctl_group_submit(struct drm_device *ddev, void *data, const struct drm_panthor_queue_submit *qsubmit = &jobs_args[i]; struct drm_sched_job *job; - job = panthor_job_create(pfile, args->group_handle, qsubmit); + job = panthor_job_create(pfile, args->group_handle, qsubmit, + file->client_id); if (IS_ERR(job)) { ret = PTR_ERR(job); goto out_cleanup_submit_ctx; diff --git a/drivers/gpu/drm/panthor/panthor_mmu.c b/drivers/gpu/drm/panthor/panthor_mmu.c index 12a02e28f50f..e0c79bd2d173 100644 --- a/drivers/gpu/drm/panthor/panthor_mmu.c +++ b/drivers/gpu/drm/panthor/panthor_mmu.c @@ -2516,7 +2516,7 @@ panthor_vm_bind_job_create(struct drm_file *file, kref_init(&job->refcount); job->vm = panthor_vm_get(vm); - ret = drm_sched_job_init(&job->base, &vm->entity, 1, vm); + ret = drm_sched_job_init(&job->base, &vm->entity, 1, vm, file->client_id); if (ret) goto err_put_job; diff --git a/drivers/gpu/drm/panthor/panthor_sched.c b/drivers/gpu/drm/panthor/panthor_sched.c index 446ec780eb4a..2af860c9068a 100644 --- a/drivers/gpu/drm/panthor/panthor_sched.c +++ b/drivers/gpu/drm/panthor/panthor_sched.c @@ -3729,7 +3729,8 @@ struct panthor_vm *panthor_job_vm(struct drm_sched_job *sched_job) struct drm_sched_job * panthor_job_create(struct panthor_file *pfile, u16 group_handle, - const struct drm_panthor_queue_submit *qsubmit) + const struct drm_panthor_queue_submit *qsubmit, + u64 drm_client_id) { struct panthor_group_pool *gpool = pfile->groups; struct panthor_job *job; @@ -3801,7 +3802,7 @@ panthor_job_create(struct panthor_file *pfile, ret = drm_sched_job_init(&job->base, &job->group->queues[job->queue_idx]->entity, - credits, job->group); + credits, job->group, drm_client_id); if (ret) goto err_put_job; diff --git a/drivers/gpu/drm/panthor/panthor_sched.h b/drivers/gpu/drm/panthor/panthor_sched.h index e650a445cf50..742b0b4ff3a3 100644 --- a/drivers/gpu/drm/panthor/panthor_sched.h +++ b/drivers/gpu/drm/panthor/panthor_sched.h @@ -29,7 +29,8 @@ int panthor_group_get_state(struct panthor_file *pfile, struct drm_sched_job * panthor_job_create(struct panthor_file *pfile, u16 group_handle, - const struct drm_panthor_queue_submit *qsubmit); + const struct drm_panthor_queue_submit *qsubmit, + u64 drm_client_id); struct drm_sched_job *panthor_job_get(struct drm_sched_job *job); struct panthor_vm *panthor_job_vm(struct drm_sched_job *sched_job); void panthor_job_put(struct drm_sched_job *job); diff --git a/drivers/gpu/drm/scheduler/sched_fence.c b/drivers/gpu/drm/scheduler/sched_fence.c index e971528504a5..d208d384d38d 100644 --- a/drivers/gpu/drm/scheduler/sched_fence.c +++ b/drivers/gpu/drm/scheduler/sched_fence.c @@ -206,7 +206,8 @@ struct drm_sched_fence *to_drm_sched_fence(struct dma_fence *f) EXPORT_SYMBOL(to_drm_sched_fence); struct drm_sched_fence *drm_sched_fence_alloc(struct drm_sched_entity *entity, - void *owner) + void *owner, + u64 drm_client_id) { struct drm_sched_fence *fence = NULL; @@ -215,6 +216,7 @@ struct drm_sched_fence *drm_sched_fence_alloc(struct drm_sched_entity *entity, return NULL; fence->owner = owner; + fence->drm_client_id = drm_client_id; spin_lock_init(&fence->lock); return fence; diff --git a/drivers/gpu/drm/scheduler/sched_internal.h b/drivers/gpu/drm/scheduler/sched_internal.h index 599cf6e1bb74..7ea5a6736f98 100644 --- a/drivers/gpu/drm/scheduler/sched_internal.h +++ b/drivers/gpu/drm/scheduler/sched_internal.h @@ -24,7 +24,7 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity); struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity); struct drm_sched_fence *drm_sched_fence_alloc(struct drm_sched_entity *s_entity, - void *owner); + void *owner, u64 drm_client_id); void drm_sched_fence_init(struct drm_sched_fence *fence, struct drm_sched_entity *entity); void drm_sched_fence_free(struct drm_sched_fence *fence); diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 829579c41c6b..9cd3b591f3c2 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -764,6 +764,8 @@ EXPORT_SYMBOL(drm_sched_resubmit_jobs); * @credits: the number of credits this job contributes to the schedulers * credit limit * @owner: job owner for debugging + * @drm_client_id: &struct drm_file.client_id of the owner (used by trace + * events) * * Refer to drm_sched_entity_push_job() documentation * for locking considerations. @@ -784,7 +786,8 @@ EXPORT_SYMBOL(drm_sched_resubmit_jobs); */ int drm_sched_job_init(struct drm_sched_job *job, struct drm_sched_entity *entity, - u32 credits, void *owner) + u32 credits, void *owner, + uint64_t drm_client_id) { if (!entity->rq) { /* This will most likely be followed by missing frames @@ -810,7 +813,7 @@ int drm_sched_job_init(struct drm_sched_job *job, job->entity = entity; job->credits = credits; - job->s_fence = drm_sched_fence_alloc(entity, owner); + job->s_fence = drm_sched_fence_alloc(entity, owner, drm_client_id); if (!job->s_fence) return -ENOMEM; diff --git a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c index f999c8859cf7..09ffbdb32d76 100644 --- a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c +++ b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c @@ -35,7 +35,7 @@ drm_mock_sched_entity_new(struct kunit *test, ret = drm_sched_entity_init(&entity->base, priority, &drm_sched, 1, - NULL); + NULL, 1); KUNIT_ASSERT_EQ(test, ret, 0); entity->test = test; diff --git a/drivers/gpu/drm/v3d/v3d_submit.c b/drivers/gpu/drm/v3d/v3d_submit.c index 4ff5de46fb22..5171ffe9012d 100644 --- a/drivers/gpu/drm/v3d/v3d_submit.c +++ b/drivers/gpu/drm/v3d/v3d_submit.c @@ -169,7 +169,7 @@ v3d_job_init(struct v3d_dev *v3d, struct drm_file *file_priv, job->file = file_priv; ret = drm_sched_job_init(&job->base, &v3d_priv->sched_entity[queue], - 1, v3d_priv); + 1, v3d_priv, file_priv->client_id); if (ret) return ret; diff --git a/drivers/gpu/drm/xe/xe_sched_job.c b/drivers/gpu/drm/xe/xe_sched_job.c index 1905ca590965..f4679cb9a56b 100644 --- a/drivers/gpu/drm/xe/xe_sched_job.c +++ b/drivers/gpu/drm/xe/xe_sched_job.c @@ -113,7 +113,8 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q, kref_init(&job->refcount); xe_exec_queue_get(job->q); - err = drm_sched_job_init(&job->drm, q->entity, 1, NULL); + err = drm_sched_job_init(&job->drm, q->entity, 1, NULL, + q->xef->drm->client_id); if (err) goto err_free; diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 1a7e377d4cbb..6fe3b4c0cffb 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -305,6 +305,13 @@ struct drm_sched_fence { * @owner: job owner for debugging */ void *owner; + + /** + * @drm_client_id: + * + * The client_id of the drm_file which owns the job. + */ + uint64_t drm_client_id; }; struct drm_sched_fence *to_drm_sched_fence(struct dma_fence *f); @@ -629,7 +636,8 @@ drm_sched_pick_best(struct drm_gpu_scheduler **sched_list, int drm_sched_job_init(struct drm_sched_job *job, struct drm_sched_entity *entity, - u32 credits, void *owner); + u32 credits, void *owner, + u64 drm_client_id); void drm_sched_job_arm(struct drm_sched_job *job); void drm_sched_entity_push_job(struct drm_sched_job *sched_job); int drm_sched_job_add_dependency(struct drm_sched_job *job, -- 2.43.0 From mcanal at igalia.com Sat May 24 13:33:48 2025 From: mcanal at igalia.com (=?UTF-8?Q?Ma=C3=ADra_Canal?=) Date: Sat, 24 May 2025 10:33:48 -0300 Subject: [PATCH 1/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: <4242fd242c7e16d0ecdf11c5d0ad795efda727a5.camel@mailbox.org> References: <20250503-sched-skip-reset-v1-0-ed0d6701a3fe@igalia.com> <20250503-sched-skip-reset-v1-1-ed0d6701a3fe@igalia.com> <4242fd242c7e16d0ecdf11c5d0ad795efda727a5.camel@mailbox.org> Message-ID: <791df72e-ff87-42e3-a4fc-527aa693a155@igalia.com> Hi Philipp, Sorry, I was OoO for a couple of weeks. On 13/05/25 04:26, Philipp Stanner wrote: > On Sat, 2025-05-03 at 17:59 -0300, Ma?ra Canal wrote: >> When the DRM scheduler times out, it's possible that the GPU isn't >> hung; >> instead, a job may still be running, and there may be no valid reason >> to >> reset the hardware. This can occur in two situations: >> >> ? 1. The GPU exposes some mechanism that ensures the GPU is still >> making >> ???? progress. By checking this mechanism, we can safely skip the >> reset, >> ???? rearm the timeout, and allow the job to continue running until >> ???? completion. This is the case for v3d and Etnaviv. >> ? 2. TDR has fired before the IRQ that signals the fence. >> Consequently, >> ???? the job actually finishes, but it triggers a timeout before >> signaling >> ???? the completion fence. >> >> These two scenarios are problematic because we remove the job from >> the >> `sched->pending_list` before calling `sched->ops->timedout_job()`. >> This >> means that when the job finally signals completion (e.g. in the IRQ >> handler), the scheduler won't call `sched->ops->free_job()`. As a >> result, >> the job and its resources won't be freed, leading to a memory leak. > > We have discussed this and discovered another, related issue. See > below. > >> >> To resolve this issue, we create a new `drm_gpu_sched_stat` that >> allows a >> driver to skip the reset. This new status will indicate that the job >> should be reinserted into the pending list, and the driver will still >> signal its completion. >> >> Signed-off-by: Ma?ra Canal >> --- >> ?drivers/gpu/drm/scheduler/sched_main.c | 14 ++++++++++++++ >> ?include/drm/gpu_scheduler.h??????????? |? 2 ++ >> ?2 files changed, 16 insertions(+) >> >> diff --git a/drivers/gpu/drm/scheduler/sched_main.c >> b/drivers/gpu/drm/scheduler/sched_main.c >> index >> 829579c41c6b5d8b2abce5ad373c7017469b7680..68ca827d77e32187a034309f881 >> 135dbc639a9b4 100644 >> --- a/drivers/gpu/drm/scheduler/sched_main.c >> +++ b/drivers/gpu/drm/scheduler/sched_main.c >> @@ -568,6 +568,17 @@ static void drm_sched_job_timedout(struct >> work_struct *work) > > So, the fundamental design problem we have is that the scheduler > assumes that when a timeout occurs, the GPU is completely hung. Your > patch addresses another aspect of that very problem. > > But if the GPU is not hung, it can signal the hardware fence at any > moment. So that's racy. Unfortunately, this already happens, which would be the second point of the list in the commit message. > > It could, theoretically, lead to backend_ops.timedout_job() being > called with a signaled job, i.e., a job that is not really timed out. > > Would you say this is *the same* issue you're describing, or a separate > one? It seems to me that it's a separate one. It isn't the issue that I'm describing in the sense that the scheduler itself won't do anything to address this issue. However, several drivers already handle with this situation internally by checking the result of `dma_fence_is_signaled()` and bailing out of the timeout if the job is signaled. We would provide a proper status code for those drivers and avoid memory leaks. > > Anyways. What I propose is that we wait until your series here has been > merged. Once that's done, we should document that drivers should expect > that backend_ops.timedout_job() can get called with a job that has not > actually timed out, and tell the scheduler about it through > DRM_GPU_SCHED_STAT_NOT_HANGING. Then the scheduler reverts the > timeout's actions, as you propose here. > > >> ? job->sched->ops->free_job(job); >> ? sched->free_guilty = false; >> ? } >> + >> + /* >> + * If the driver indicated that the GPU is still >> running and wants to skip >> + * the reset, reinsert the job back into the pending >> list and realarm the >> + * timeout. >> + */ >> + if (status == DRM_GPU_SCHED_STAT_RUNNING) { >> + spin_lock(&sched->job_list_lock); >> + list_add(&job->list, &sched->pending_list); >> + spin_unlock(&sched->job_list_lock); >> + } > > btw, if you go for Matt's requeue work item approach, it'll be better > to write a helper function with a clear name for all that. > > drm_sched_job_reinsert_on_false_timout() maybe. Thanks for the review, I'll send v2 soon. Best Regards, - Ma?ra > > > P. > > >> ? } else { >> ? spin_unlock(&sched->job_list_lock); >> ? } From pierre-eric.pelloux-prayer at amd.com Mon May 26 12:54:42 2025 From: pierre-eric.pelloux-prayer at amd.com (Pierre-Eric Pelloux-Prayer) Date: Mon, 26 May 2025 14:54:42 +0200 Subject: [PATCH v11 00/10] Improve gpu_scheduler trace events + UAPI Message-ID: <20250526125505.2360-1-pierre-eric.pelloux-prayer@amd.com> Hi, The initial goal of this series was to improve the drm and amdgpu trace events to be able to expose more of the inner workings of the scheduler and drivers to developers via tools. Then, the series evolved to become focused only on gpu_scheduler. The changes around vblank events will be part of a different series, as well as the amdgpu ones. Moreover Sima suggested to make some trace events stable uAPI, so tools can rely on them long term. The first patches extend and cleanup the gpu scheduler events, then add a documentation entry in drm-uapi.rst. The last 2 patches are new in v8. One is based on a suggestion from Tvrtko and gets rid of drm_sched_job::id. The other is a cleanup of amdgpu trace events to use the fence=%llu:%llu format. The drm_sched_job patches don't affect gpuvis which has code to parse the gpu_scheduler events but these events are not enabled. Changes since v10: * fixed 2 errors reported by kernel test robot * rebased on drm-misc-next Changes since v9: * fixed documentation link syntax * fixed typos in commit messages * spelled out that these events cannot be used before drm_sched_job_arm has been called Changes since v8: * swapped patches 8 & 9 * rebased on drm-next Changes since v7: * uint64_t -> u64 * reworked dependencies tracing (Tvrtko) * use common name prefix for all events (Tvrtko) * dropped drm_sched_job::id (Tvrtko) Useful links: - userspace tool using the updated events: https://gitlab.freedesktop.org/tomstdenis/umr/-/merge_requests/37 - v8: https://lists.freedesktop.org/archives/dri-devel/2025-March/496781.html Pierre-Eric Pelloux-Prayer (10): drm/debugfs: Output client_id in in drm_clients_info drm/sched: Store the drm client_id in drm_sched_fence drm/sched: Add device name to the drm_sched_process_job event drm/sched: Cleanup gpu_scheduler trace events drm/sched: Trace dependencies for GPU jobs drm/sched: Add the drm_client_id to the drm_sched_run/exec_job events drm/sched: Cleanup event names drm: Get rid of drm_sched_job.id drm/doc: Document some tracepoints as uAPI drm/amdgpu: update trace format to match gpu_scheduler_trace Documentation/gpu/drm-uapi.rst | 19 ++++ drivers/accel/amdxdna/aie2_ctx.c | 3 +- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 3 +- drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 8 +- drivers/gpu/drm/amd/amdgpu/amdgpu_job.h | 3 +- drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h | 32 ++---- drivers/gpu/drm/drm_debugfs.c | 10 +- drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c | 2 +- drivers/gpu/drm/imagination/pvr_job.c | 2 +- drivers/gpu/drm/imagination/pvr_queue.c | 5 +- drivers/gpu/drm/imagination/pvr_queue.h | 2 +- drivers/gpu/drm/lima/lima_gem.c | 2 +- drivers/gpu/drm/lima/lima_sched.c | 6 +- drivers/gpu/drm/lima/lima_sched.h | 3 +- drivers/gpu/drm/lima/lima_trace.h | 6 +- drivers/gpu/drm/msm/msm_gem_submit.c | 8 +- drivers/gpu/drm/nouveau/nouveau_sched.c | 3 +- drivers/gpu/drm/panfrost/panfrost_drv.c | 2 +- drivers/gpu/drm/panthor/panthor_drv.c | 3 +- drivers/gpu/drm/panthor/panthor_mmu.c | 2 +- drivers/gpu/drm/panthor/panthor_sched.c | 5 +- drivers/gpu/drm/panthor/panthor_sched.h | 3 +- .../gpu/drm/scheduler/gpu_scheduler_trace.h | 103 +++++++++++++----- drivers/gpu/drm/scheduler/sched_entity.c | 16 ++- drivers/gpu/drm/scheduler/sched_fence.c | 4 +- drivers/gpu/drm/scheduler/sched_internal.h | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 12 +- .../gpu/drm/scheduler/tests/mock_scheduler.c | 3 +- drivers/gpu/drm/v3d/v3d_submit.c | 2 +- drivers/gpu/drm/xe/xe_sched_job.c | 3 +- include/drm/gpu_scheduler.h | 13 ++- 32 files changed, 191 insertions(+), 101 deletions(-) -- 2.43.0 From pierre-eric.pelloux-prayer at amd.com Mon May 26 12:54:44 2025 From: pierre-eric.pelloux-prayer at amd.com (Pierre-Eric Pelloux-Prayer) Date: Mon, 26 May 2025 14:54:44 +0200 Subject: [PATCH v11 02/10] drm/sched: Store the drm client_id in drm_sched_fence In-Reply-To: <20250526125505.2360-1-pierre-eric.pelloux-prayer@amd.com> References: <20250526125505.2360-1-pierre-eric.pelloux-prayer@amd.com> Message-ID: <20250526125505.2360-3-pierre-eric.pelloux-prayer@amd.com> This will be used in a later commit to trace the drm client_id in some of the gpu_scheduler trace events. This requires changing all the users of drm_sched_job_init to add an extra parameter. The newly added drm_client_id field in the drm_sched_fence is a bit of a duplicate of the owner one. One suggestion I received was to merge those 2 fields - this can't be done right now as amdgpu uses some special values (AMDGPU_FENCE_OWNER_*) that can't really be translated into a client id. Christian is working on getting rid of those; when it's done we should be able to squash owner/drm_client_id together. Reviewed-by: Christian K?nig Signed-off-by: Pierre-Eric Pelloux-Prayer --- drivers/accel/amdxdna/aie2_ctx.c | 3 ++- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 3 ++- drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 8 +++++--- drivers/gpu/drm/amd/amdgpu/amdgpu_job.h | 3 ++- drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c | 2 +- drivers/gpu/drm/imagination/pvr_job.c | 2 +- drivers/gpu/drm/imagination/pvr_queue.c | 5 +++-- drivers/gpu/drm/imagination/pvr_queue.h | 2 +- drivers/gpu/drm/lima/lima_gem.c | 2 +- drivers/gpu/drm/lima/lima_sched.c | 6 ++++-- drivers/gpu/drm/lima/lima_sched.h | 3 ++- drivers/gpu/drm/msm/msm_gem_submit.c | 8 +++++--- drivers/gpu/drm/nouveau/nouveau_sched.c | 3 ++- drivers/gpu/drm/panfrost/panfrost_drv.c | 2 +- drivers/gpu/drm/panthor/panthor_drv.c | 3 ++- drivers/gpu/drm/panthor/panthor_mmu.c | 2 +- drivers/gpu/drm/panthor/panthor_sched.c | 5 +++-- drivers/gpu/drm/panthor/panthor_sched.h | 3 ++- drivers/gpu/drm/scheduler/sched_fence.c | 4 +++- drivers/gpu/drm/scheduler/sched_internal.h | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 7 +++++-- drivers/gpu/drm/scheduler/tests/mock_scheduler.c | 3 ++- drivers/gpu/drm/v3d/v3d_submit.c | 2 +- drivers/gpu/drm/xe/xe_sched_job.c | 3 ++- include/drm/gpu_scheduler.h | 10 +++++++++- 26 files changed, 64 insertions(+), 34 deletions(-) diff --git a/drivers/accel/amdxdna/aie2_ctx.c b/drivers/accel/amdxdna/aie2_ctx.c index e04549f64d69..3e38a5f637ea 100644 --- a/drivers/accel/amdxdna/aie2_ctx.c +++ b/drivers/accel/amdxdna/aie2_ctx.c @@ -848,7 +848,8 @@ int aie2_cmd_submit(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, goto up_sem; } - ret = drm_sched_job_init(&job->base, &hwctx->priv->entity, 1, hwctx); + ret = drm_sched_job_init(&job->base, &hwctx->priv->entity, 1, hwctx, + hwctx->client->filp->client_id); if (ret) { XDNA_ERR(xdna, "DRM job init failed, ret %d", ret); goto free_chain; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c index 4cec3a873995..1a77ba7036c9 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c @@ -639,7 +639,7 @@ int amdgpu_amdkfd_submit_ib(struct amdgpu_device *adev, goto err; } - ret = amdgpu_job_alloc(adev, NULL, NULL, NULL, 1, &job); + ret = amdgpu_job_alloc(adev, NULL, NULL, NULL, 1, &job, 0); if (ret) goto err; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c index 82df06a72ee0..5a231b997d65 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c @@ -293,7 +293,8 @@ static int amdgpu_cs_pass1(struct amdgpu_cs_parser *p, for (i = 0; i < p->gang_size; ++i) { ret = amdgpu_job_alloc(p->adev, vm, p->entities[i], vm, - num_ibs[i], &p->jobs[i]); + num_ibs[i], &p->jobs[i], + p->filp->client_id); if (ret) goto free_all_kdata; p->jobs[i]->enforce_isolation = p->adev->enforce_isolation[fpriv->xcp_id]; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c index acb21fc8b3ce..75262ce8db27 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c @@ -204,7 +204,8 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job) int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm *vm, struct drm_sched_entity *entity, void *owner, - unsigned int num_ibs, struct amdgpu_job **job) + unsigned int num_ibs, struct amdgpu_job **job, + u64 drm_client_id) { if (num_ibs == 0) return -EINVAL; @@ -222,7 +223,8 @@ int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm *vm, if (!entity) return 0; - return drm_sched_job_init(&(*job)->base, entity, 1, owner); + return drm_sched_job_init(&(*job)->base, entity, 1, owner, + drm_client_id); } int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, @@ -232,7 +234,7 @@ int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, { int r; - r = amdgpu_job_alloc(adev, NULL, entity, owner, 1, job); + r = amdgpu_job_alloc(adev, NULL, entity, owner, 1, job, 0); if (r) return r; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h index ce6b9ba967ff..5a8bc6342222 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h @@ -90,7 +90,8 @@ static inline struct amdgpu_ring *amdgpu_job_ring(struct amdgpu_job *job) int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm *vm, struct drm_sched_entity *entity, void *owner, - unsigned int num_ibs, struct amdgpu_job **job); + unsigned int num_ibs, struct amdgpu_job **job, + u64 drm_client_id); int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, struct drm_sched_entity *entity, void *owner, size_t size, enum amdgpu_ib_pool_type pool_type, diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c index 3c0a5c3e0e3d..76c742328edb 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c @@ -534,7 +534,7 @@ int etnaviv_ioctl_gem_submit(struct drm_device *dev, void *data, ret = drm_sched_job_init(&submit->sched_job, &ctx->sched_entity[args->pipe], - 1, submit->ctx); + 1, submit->ctx, file->client_id); if (ret) goto err_submit_put; diff --git a/drivers/gpu/drm/imagination/pvr_job.c b/drivers/gpu/drm/imagination/pvr_job.c index 59b334d094fa..7564b0f21b42 100644 --- a/drivers/gpu/drm/imagination/pvr_job.c +++ b/drivers/gpu/drm/imagination/pvr_job.c @@ -446,7 +446,7 @@ create_job(struct pvr_device *pvr_dev, if (err) goto err_put_job; - err = pvr_queue_job_init(job); + err = pvr_queue_job_init(job, pvr_file->file->client_id); if (err) goto err_put_job; diff --git a/drivers/gpu/drm/imagination/pvr_queue.c b/drivers/gpu/drm/imagination/pvr_queue.c index 5e9bc0992824..5a41ee79fed6 100644 --- a/drivers/gpu/drm/imagination/pvr_queue.c +++ b/drivers/gpu/drm/imagination/pvr_queue.c @@ -1073,6 +1073,7 @@ static int pvr_queue_cleanup_fw_context(struct pvr_queue *queue) /** * pvr_queue_job_init() - Initialize queue related fields in a pvr_job object. * @job: The job to initialize. + * @drm_client_id: drm_file.client_id submitting the job * * Bind the job to a queue and allocate memory to guarantee pvr_queue_job_arm() * and pvr_queue_job_push() can't fail. We also make sure the context type is @@ -1082,7 +1083,7 @@ static int pvr_queue_cleanup_fw_context(struct pvr_queue *queue) * * 0 on success, or * * An error code if something failed. */ -int pvr_queue_job_init(struct pvr_job *job) +int pvr_queue_job_init(struct pvr_job *job, u64 drm_client_id) { /* Fragment jobs need at least one native fence wait on the geometry job fence. */ u32 min_native_dep_count = job->type == DRM_PVR_JOB_TYPE_FRAGMENT ? 1 : 0; @@ -1099,7 +1100,7 @@ int pvr_queue_job_init(struct pvr_job *job) if (!pvr_cccb_cmdseq_can_fit(&queue->cccb, job_cmds_size(job, min_native_dep_count))) return -E2BIG; - err = drm_sched_job_init(&job->base, &queue->entity, 1, THIS_MODULE); + err = drm_sched_job_init(&job->base, &queue->entity, 1, THIS_MODULE, drm_client_id); if (err) return err; diff --git a/drivers/gpu/drm/imagination/pvr_queue.h b/drivers/gpu/drm/imagination/pvr_queue.h index 93fe9ac9f58c..fc1986d73fc8 100644 --- a/drivers/gpu/drm/imagination/pvr_queue.h +++ b/drivers/gpu/drm/imagination/pvr_queue.h @@ -143,7 +143,7 @@ struct pvr_queue { bool pvr_queue_fence_is_ufo_backed(struct dma_fence *f); -int pvr_queue_job_init(struct pvr_job *job); +int pvr_queue_job_init(struct pvr_job *job, u64 drm_client_id); void pvr_queue_job_cleanup(struct pvr_job *job); diff --git a/drivers/gpu/drm/lima/lima_gem.c b/drivers/gpu/drm/lima/lima_gem.c index 5deec673c11e..9722b847a539 100644 --- a/drivers/gpu/drm/lima/lima_gem.c +++ b/drivers/gpu/drm/lima/lima_gem.c @@ -341,7 +341,7 @@ int lima_gem_submit(struct drm_file *file, struct lima_submit *submit) err = lima_sched_task_init( submit->task, submit->ctx->context + submit->pipe, - bos, submit->nr_bos, vm); + bos, submit->nr_bos, vm, file->client_id); if (err) goto err_out1; diff --git a/drivers/gpu/drm/lima/lima_sched.c b/drivers/gpu/drm/lima/lima_sched.c index 7934098e651b..954f4325b859 100644 --- a/drivers/gpu/drm/lima/lima_sched.c +++ b/drivers/gpu/drm/lima/lima_sched.c @@ -113,7 +113,8 @@ static inline struct lima_sched_pipe *to_lima_pipe(struct drm_gpu_scheduler *sch int lima_sched_task_init(struct lima_sched_task *task, struct lima_sched_context *context, struct lima_bo **bos, int num_bos, - struct lima_vm *vm) + struct lima_vm *vm, + u64 drm_client_id) { int err, i; @@ -124,7 +125,8 @@ int lima_sched_task_init(struct lima_sched_task *task, for (i = 0; i < num_bos; i++) drm_gem_object_get(&bos[i]->base.base); - err = drm_sched_job_init(&task->base, &context->base, 1, vm); + err = drm_sched_job_init(&task->base, &context->base, 1, vm, + drm_client_id); if (err) { kfree(task->bos); return err; diff --git a/drivers/gpu/drm/lima/lima_sched.h b/drivers/gpu/drm/lima/lima_sched.h index 85b23ba901d5..1a08faf8a529 100644 --- a/drivers/gpu/drm/lima/lima_sched.h +++ b/drivers/gpu/drm/lima/lima_sched.h @@ -88,7 +88,8 @@ struct lima_sched_pipe { int lima_sched_task_init(struct lima_sched_task *task, struct lima_sched_context *context, struct lima_bo **bos, int num_bos, - struct lima_vm *vm); + struct lima_vm *vm, + u64 drm_client_id); void lima_sched_task_fini(struct lima_sched_task *task); int lima_sched_context_init(struct lima_sched_pipe *pipe, diff --git a/drivers/gpu/drm/msm/msm_gem_submit.c b/drivers/gpu/drm/msm/msm_gem_submit.c index 3e9aa2cc38ef..d9be0fe3d674 100644 --- a/drivers/gpu/drm/msm/msm_gem_submit.c +++ b/drivers/gpu/drm/msm/msm_gem_submit.c @@ -30,7 +30,7 @@ static struct msm_gem_submit *submit_create(struct drm_device *dev, struct msm_gpu *gpu, struct msm_gpu_submitqueue *queue, uint32_t nr_bos, - uint32_t nr_cmds) + uint32_t nr_cmds, u64 drm_client_id) { static atomic_t ident = ATOMIC_INIT(0); struct msm_gem_submit *submit; @@ -54,7 +54,8 @@ static struct msm_gem_submit *submit_create(struct drm_device *dev, return ERR_PTR(ret); } - ret = drm_sched_job_init(&submit->base, queue->entity, 1, queue); + ret = drm_sched_job_init(&submit->base, queue->entity, 1, queue, + drm_client_id); if (ret) { kfree(submit->hw_fence); kfree(submit); @@ -693,7 +694,8 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data, } } - submit = submit_create(dev, gpu, queue, args->nr_bos, args->nr_cmds); + submit = submit_create(dev, gpu, queue, args->nr_bos, args->nr_cmds, + file->client_id); if (IS_ERR(submit)) { ret = PTR_ERR(submit); goto out_post_unlock; diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.c b/drivers/gpu/drm/nouveau/nouveau_sched.c index d326e55d2d24..460a5fb02412 100644 --- a/drivers/gpu/drm/nouveau/nouveau_sched.c +++ b/drivers/gpu/drm/nouveau/nouveau_sched.c @@ -87,7 +87,8 @@ nouveau_job_init(struct nouveau_job *job, } ret = drm_sched_job_init(&job->base, &sched->entity, - args->credits, NULL); + args->credits, NULL, + job->file_priv->client_id); if (ret) goto err_free_chains; diff --git a/drivers/gpu/drm/panfrost/panfrost_drv.c b/drivers/gpu/drm/panfrost/panfrost_drv.c index f1ec3b02f15a..50d8d16eace0 100644 --- a/drivers/gpu/drm/panfrost/panfrost_drv.c +++ b/drivers/gpu/drm/panfrost/panfrost_drv.c @@ -312,7 +312,7 @@ static int panfrost_ioctl_submit(struct drm_device *dev, void *data, ret = drm_sched_job_init(&job->base, &file_priv->sched_entity[slot], - 1, NULL); + 1, NULL, file->client_id); if (ret) goto out_put_job; diff --git a/drivers/gpu/drm/panthor/panthor_drv.c b/drivers/gpu/drm/panthor/panthor_drv.c index 6200cad22563..229b9190f152 100644 --- a/drivers/gpu/drm/panthor/panthor_drv.c +++ b/drivers/gpu/drm/panthor/panthor_drv.c @@ -996,7 +996,8 @@ static int panthor_ioctl_group_submit(struct drm_device *ddev, void *data, const struct drm_panthor_queue_submit *qsubmit = &jobs_args[i]; struct drm_sched_job *job; - job = panthor_job_create(pfile, args->group_handle, qsubmit); + job = panthor_job_create(pfile, args->group_handle, qsubmit, + file->client_id); if (IS_ERR(job)) { ret = PTR_ERR(job); goto out_cleanup_submit_ctx; diff --git a/drivers/gpu/drm/panthor/panthor_mmu.c b/drivers/gpu/drm/panthor/panthor_mmu.c index 6ca9a2642a4e..4ae72b211793 100644 --- a/drivers/gpu/drm/panthor/panthor_mmu.c +++ b/drivers/gpu/drm/panthor/panthor_mmu.c @@ -2523,7 +2523,7 @@ panthor_vm_bind_job_create(struct drm_file *file, kref_init(&job->refcount); job->vm = panthor_vm_get(vm); - ret = drm_sched_job_init(&job->base, &vm->entity, 1, vm); + ret = drm_sched_job_init(&job->base, &vm->entity, 1, vm, file->client_id); if (ret) goto err_put_job; diff --git a/drivers/gpu/drm/panthor/panthor_sched.c b/drivers/gpu/drm/panthor/panthor_sched.c index 43ee57728de5..a2248f692a03 100644 --- a/drivers/gpu/drm/panthor/panthor_sched.c +++ b/drivers/gpu/drm/panthor/panthor_sched.c @@ -3732,7 +3732,8 @@ struct panthor_vm *panthor_job_vm(struct drm_sched_job *sched_job) struct drm_sched_job * panthor_job_create(struct panthor_file *pfile, u16 group_handle, - const struct drm_panthor_queue_submit *qsubmit) + const struct drm_panthor_queue_submit *qsubmit, + u64 drm_client_id) { struct panthor_group_pool *gpool = pfile->groups; struct panthor_job *job; @@ -3804,7 +3805,7 @@ panthor_job_create(struct panthor_file *pfile, ret = drm_sched_job_init(&job->base, &job->group->queues[job->queue_idx]->entity, - credits, job->group); + credits, job->group, drm_client_id); if (ret) goto err_put_job; diff --git a/drivers/gpu/drm/panthor/panthor_sched.h b/drivers/gpu/drm/panthor/panthor_sched.h index e650a445cf50..742b0b4ff3a3 100644 --- a/drivers/gpu/drm/panthor/panthor_sched.h +++ b/drivers/gpu/drm/panthor/panthor_sched.h @@ -29,7 +29,8 @@ int panthor_group_get_state(struct panthor_file *pfile, struct drm_sched_job * panthor_job_create(struct panthor_file *pfile, u16 group_handle, - const struct drm_panthor_queue_submit *qsubmit); + const struct drm_panthor_queue_submit *qsubmit, + u64 drm_client_id); struct drm_sched_job *panthor_job_get(struct drm_sched_job *job); struct panthor_vm *panthor_job_vm(struct drm_sched_job *sched_job); void panthor_job_put(struct drm_sched_job *job); diff --git a/drivers/gpu/drm/scheduler/sched_fence.c b/drivers/gpu/drm/scheduler/sched_fence.c index d6239e015b66..725de257d60d 100644 --- a/drivers/gpu/drm/scheduler/sched_fence.c +++ b/drivers/gpu/drm/scheduler/sched_fence.c @@ -205,7 +205,8 @@ struct drm_sched_fence *to_drm_sched_fence(struct dma_fence *f) EXPORT_SYMBOL(to_drm_sched_fence); struct drm_sched_fence *drm_sched_fence_alloc(struct drm_sched_entity *entity, - void *owner) + void *owner, + u64 drm_client_id) { struct drm_sched_fence *fence = NULL; @@ -214,6 +215,7 @@ struct drm_sched_fence *drm_sched_fence_alloc(struct drm_sched_entity *entity, return NULL; fence->owner = owner; + fence->drm_client_id = drm_client_id; spin_lock_init(&fence->lock); return fence; diff --git a/drivers/gpu/drm/scheduler/sched_internal.h b/drivers/gpu/drm/scheduler/sched_internal.h index 599cf6e1bb74..7ea5a6736f98 100644 --- a/drivers/gpu/drm/scheduler/sched_internal.h +++ b/drivers/gpu/drm/scheduler/sched_internal.h @@ -24,7 +24,7 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity); struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity); struct drm_sched_fence *drm_sched_fence_alloc(struct drm_sched_entity *s_entity, - void *owner); + void *owner, u64 drm_client_id); void drm_sched_fence_init(struct drm_sched_fence *fence, struct drm_sched_entity *entity); void drm_sched_fence_free(struct drm_sched_fence *fence); diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index f7118497e47a..34252bac83b9 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -766,6 +766,8 @@ EXPORT_SYMBOL(drm_sched_resubmit_jobs); * @credits: the number of credits this job contributes to the schedulers * credit limit * @owner: job owner for debugging + * @drm_client_id: &struct drm_file.client_id of the owner (used by trace + * events) * * Refer to drm_sched_entity_push_job() documentation * for locking considerations. @@ -786,7 +788,8 @@ EXPORT_SYMBOL(drm_sched_resubmit_jobs); */ int drm_sched_job_init(struct drm_sched_job *job, struct drm_sched_entity *entity, - u32 credits, void *owner) + u32 credits, void *owner, + uint64_t drm_client_id) { if (!entity->rq) { /* This will most likely be followed by missing frames @@ -812,7 +815,7 @@ int drm_sched_job_init(struct drm_sched_job *job, job->entity = entity; job->credits = credits; - job->s_fence = drm_sched_fence_alloc(entity, owner); + job->s_fence = drm_sched_fence_alloc(entity, owner, drm_client_id); if (!job->s_fence) return -ENOMEM; diff --git a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c index f999c8859cf7..fcddaeaa9217 100644 --- a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c +++ b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c @@ -117,7 +117,8 @@ drm_mock_sched_job_new(struct kunit *test, ret = drm_sched_job_init(&job->base, &entity->base, 1, - NULL); + NULL, + 1); KUNIT_ASSERT_EQ(test, ret, 0); job->test = test; diff --git a/drivers/gpu/drm/v3d/v3d_submit.c b/drivers/gpu/drm/v3d/v3d_submit.c index 4ff5de46fb22..5171ffe9012d 100644 --- a/drivers/gpu/drm/v3d/v3d_submit.c +++ b/drivers/gpu/drm/v3d/v3d_submit.c @@ -169,7 +169,7 @@ v3d_job_init(struct v3d_dev *v3d, struct drm_file *file_priv, job->file = file_priv; ret = drm_sched_job_init(&job->base, &v3d_priv->sched_entity[queue], - 1, v3d_priv); + 1, v3d_priv, file_priv->client_id); if (ret) return ret; diff --git a/drivers/gpu/drm/xe/xe_sched_job.c b/drivers/gpu/drm/xe/xe_sched_job.c index f0a6ce610948..5921293b25db 100644 --- a/drivers/gpu/drm/xe/xe_sched_job.c +++ b/drivers/gpu/drm/xe/xe_sched_job.c @@ -113,7 +113,8 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q, kref_init(&job->refcount); xe_exec_queue_get(job->q); - err = drm_sched_job_init(&job->drm, q->entity, 1, NULL); + err = drm_sched_job_init(&job->drm, q->entity, 1, NULL, + q->xef->drm->client_id); if (err) goto err_free; diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index d860db087ea5..1270cd82ff3e 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -305,6 +305,13 @@ struct drm_sched_fence { * @owner: job owner for debugging */ void *owner; + + /** + * @drm_client_id: + * + * The client_id of the drm_file which owns the job. + */ + uint64_t drm_client_id; }; struct drm_sched_fence *to_drm_sched_fence(struct dma_fence *f); @@ -629,7 +636,8 @@ drm_sched_pick_best(struct drm_gpu_scheduler **sched_list, int drm_sched_job_init(struct drm_sched_job *job, struct drm_sched_entity *entity, - u32 credits, void *owner); + u32 credits, void *owner, + u64 drm_client_id); void drm_sched_job_arm(struct drm_sched_job *job); void drm_sched_entity_push_job(struct drm_sched_job *sched_job); int drm_sched_job_add_dependency(struct drm_sched_job *job, -- 2.43.0 From phasta at mailbox.org Wed May 28 14:18:30 2025 From: phasta at mailbox.org (Philipp Stanner) Date: Wed, 28 May 2025 16:18:30 +0200 Subject: [PATCH v11 00/10] Improve gpu_scheduler trace events + UAPI In-Reply-To: <20250526125505.2360-1-pierre-eric.pelloux-prayer@amd.com> References: <20250526125505.2360-1-pierre-eric.pelloux-prayer@amd.com> Message-ID: On Mon, 2025-05-26 at 14:54 +0200, Pierre-Eric Pelloux-Prayer wrote: > Hi, > > The initial goal of this series was to improve the drm and amdgpu > trace events to be able to expose more of the inner workings of > the scheduler and drivers to developers via tools. > > Then, the series evolved to become focused only on gpu_scheduler. > The changes around vblank events will be part of a different > series, as well as the amdgpu ones. > > Moreover Sima suggested to make some trace events stable uAPI, > so tools can rely on them long term. > > The first patches extend and cleanup the gpu scheduler events, > then add a documentation entry in drm-uapi.rst. > > The last 2 patches are new in v8. One is based on a suggestion > from Tvrtko and gets rid of drm_sched_job::id. The other is a > cleanup of amdgpu trace events to use the fence=%llu:%llu format. > > The drm_sched_job patches don't affect gpuvis which has code to parse > the gpu_scheduler events but these events are not enabled. > > Changes since v10: > * fixed 2 errors reported by kernel test robot > * rebased on drm-misc-next > > Changes since v9: > * fixed documentation link syntax > * fixed typos in commit messages > * spelled out that these events cannot be used before > ? drm_sched_job_arm has been called > > Changes since v8: > * swapped patches 8 & 9 > * rebased on drm-next > > Changes since v7: > * uint64_t -> u64 > * reworked dependencies tracing (Tvrtko) > * use common name prefix for all events (Tvrtko) > * dropped drm_sched_job::id (Tvrtko) > > Useful links: > - userspace tool using the updated events: > https://gitlab.freedesktop.org/tomstdenis/umr/-/merge_requests/37 > - v8: > https://lists.freedesktop.org/archives/dri-devel/2025-March/496781.html > > Pierre-Eric Pelloux-Prayer (10): > ? drm/debugfs: Output client_id in in drm_clients_info > ? drm/sched: Store the drm client_id in drm_sched_fence > ? drm/sched: Add device name to the drm_sched_process_job event > ? drm/sched: Cleanup gpu_scheduler trace events > ? drm/sched: Trace dependencies for GPU jobs > ? drm/sched: Add the drm_client_id to the drm_sched_run/exec_job > events > ? drm/sched: Cleanup event names > ? drm: Get rid of drm_sched_job.id > ? drm/doc: Document some tracepoints as uAPI > ? drm/amdgpu: update trace format to match gpu_scheduler_trace Applied to drm-misc-next Thanks P. > > ?Documentation/gpu/drm-uapi.rst??????????????? |? 19 ++++ > ?drivers/accel/amdxdna/aie2_ctx.c????????????? |?? 3 +- > ?drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c??? |?? 2 +- > ?drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c??????? |?? 3 +- > ?drivers/gpu/drm/amd/amdgpu/amdgpu_job.c?????? |?? 8 +- > ?drivers/gpu/drm/amd/amdgpu/amdgpu_job.h?????? |?? 3 +- > ?drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h???? |? 32 ++---- > ?drivers/gpu/drm/drm_debugfs.c???????????????? |? 10 +- > ?drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c? |?? 2 +- > ?drivers/gpu/drm/imagination/pvr_job.c???????? |?? 2 +- > ?drivers/gpu/drm/imagination/pvr_queue.c?????? |?? 5 +- > ?drivers/gpu/drm/imagination/pvr_queue.h?????? |?? 2 +- > ?drivers/gpu/drm/lima/lima_gem.c?????????????? |?? 2 +- > ?drivers/gpu/drm/lima/lima_sched.c???????????? |?? 6 +- > ?drivers/gpu/drm/lima/lima_sched.h???????????? |?? 3 +- > ?drivers/gpu/drm/lima/lima_trace.h???????????? |?? 6 +- > ?drivers/gpu/drm/msm/msm_gem_submit.c????????? |?? 8 +- > ?drivers/gpu/drm/nouveau/nouveau_sched.c?????? |?? 3 +- > ?drivers/gpu/drm/panfrost/panfrost_drv.c?????? |?? 2 +- > ?drivers/gpu/drm/panthor/panthor_drv.c???????? |?? 3 +- > ?drivers/gpu/drm/panthor/panthor_mmu.c???????? |?? 2 +- > ?drivers/gpu/drm/panthor/panthor_sched.c?????? |?? 5 +- > ?drivers/gpu/drm/panthor/panthor_sched.h?????? |?? 3 +- > ?.../gpu/drm/scheduler/gpu_scheduler_trace.h?? | 103 +++++++++++++--- > -- > ?drivers/gpu/drm/scheduler/sched_entity.c????? |? 16 ++- > ?drivers/gpu/drm/scheduler/sched_fence.c?????? |?? 4 +- > ?drivers/gpu/drm/scheduler/sched_internal.h??? |?? 2 +- > ?drivers/gpu/drm/scheduler/sched_main.c??????? |? 12 +- > ?.../gpu/drm/scheduler/tests/mock_scheduler.c? |?? 3 +- > ?drivers/gpu/drm/v3d/v3d_submit.c????????????? |?? 2 +- > ?drivers/gpu/drm/xe/xe_sched_job.c???????????? |?? 3 +- > ?include/drm/gpu_scheduler.h?????????????????? |? 13 ++- > ?32 files changed, 191 insertions(+), 101 deletions(-) > From pierre-eric at damsy.net Wed May 28 20:46:19 2025 From: pierre-eric at damsy.net (Pierre-Eric Pelloux-Prayer) Date: Wed, 28 May 2025 22:46:19 +0200 Subject: [PATCH v11 02/10] drm/sched: Store the drm client_id in drm_sched_fence In-Reply-To: <5jnpsmjef5ibegbsbelkfmudv4wagpcfb25nptqs5z4ccitq4c@3bdtrbrrmtil> References: <20250526125505.2360-1-pierre-eric.pelloux-prayer@amd.com> <20250526125505.2360-3-pierre-eric.pelloux-prayer@amd.com> <5jnpsmjef5ibegbsbelkfmudv4wagpcfb25nptqs5z4ccitq4c@3bdtrbrrmtil> Message-ID: <5e312895-27ed-4ad4-b1c6-55035cdcd112@damsy.net> Hi, Le 28/05/2025 ? 21:07, Lucas De Marchi a ?crit?: > On Mon, May 26, 2025 at 02:54:44PM +0200, Pierre-Eric Pelloux-Prayer wrote: >> drivers/gpu/drm/xe/xe_sched_job.c??????????????? |? 3 ++- >> diff --git a/drivers/gpu/drm/xe/xe_sched_job.c b/drivers/gpu/drm/xe/xe_sched_job.c >> index f0a6ce610948..5921293b25db 100644 >> --- a/drivers/gpu/drm/xe/xe_sched_job.c >> +++ b/drivers/gpu/drm/xe/xe_sched_job.c >> @@ -113,7 +113,8 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q, >> ????kref_init(&job->refcount); >> ????xe_exec_queue_get(job->q); >> >> -??? err = drm_sched_job_init(&job->drm, q->entity, 1, NULL); >> +??? err = drm_sched_job_init(&job->drm, q->entity, 1, NULL, >> +???????????????? q->xef->drm->client_id); > > you can't do this here. xef is only !NULL if it's a job from userspace. sorry about that. > For in-kernel jobs, xef is NULL and this explodes. Right now this > completely breaks xe since one of the very first things we do is > to submit a job to save the default context. Example: > https://intel-gfx-ci.01.org/tree/intel-xe/xe-3151-56d2b14961751a677ff1f7ff8b93a6c814ce2be3/bat- > bmg-1/igt at xe_module_load@load.html > > ????<4> [] RIP: 0010:xe_sched_job_create+0xbd/0x390 [xe] > ????<4> [] Code: c1 43 18 85 c0 0f 84 6f 02 00 00 8d 50 01 09 c2 0f 88 3e 02 00 00 48 8b 03 48 8b > b3 d8 00 00 00 31 c9 4c 89 ef ba 01 00 00 00 <48> 8b 40 08 4c 8b 40 60 e8 86 64 7c ff 41 89 c4 85 c0 > 0f 85 9b 01 > ????<4> [] RSP: 0018:ffffc900031972d8 EFLAGS: 00010246 > ????<4> [] RAX: 0000000000000000 RBX: ffff88815fc40d00 RCX: 0000000000000000 > ????<4> [] RDX: 0000000000000001 RSI: ffff88812e6552a8 RDI: ffff88815f939c40 > ????<4> [] RBP: ffffc90003197318 R08: 0000000000000000 R09: 0000000000000000 > ????<4> [] R10: 0000000000000000 R11: 0000000000000000 R12: ffffc90003197428 > ????<4> [] R13: ffff88815f939c40 R14: ffff88811f054000 R15: ffff88815fc40d00 > ????<4> [] FS:? 00007681f2948940(0000) GS:ffff8888daf14000(0000) knlGS:0000000000000000 > ????<4> [] CS:? 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > ????<4> [] CR2: 0000000000000008 CR3: 0000000118315004 CR4: 0000000000f72ef0 > ????<4> [] PKRU: 55555554 > ????<4> [] Call Trace: > ????<4> []? > ????<4> []? __xe_bb_create_job+0xa2/0x240 [xe] > ????<4> []? ? find_held_lock+0x31/0x90 > ????<4> []? ? xa_find_after+0x12c/0x250 > ????<4> []? xe_bb_create_job+0x6e/0x380 [xe] > ????<4> []? ? xa_find_after+0x136/0x250 > ????<4> []? ? __drm_dev_dbg+0x7d/0xb0 > ????<4> []? xe_gt_record_default_lrcs+0x542/0xb00 [xe] > > Can we use 0 for in-kernel client since drm_file starts them from 1? Yes, this is what amdgpu does. > Like this: > > | diff --git a/drivers/gpu/drm/xe/xe_sched_job.c b/drivers/gpu/drm/xe/xe_sched_job.c > | index 5921293b25db3..d21bf8f269640 100644 > | --- a/drivers/gpu/drm/xe/xe_sched_job.c > | +++ b/drivers/gpu/drm/xe/xe_sched_job.c > | @@ -114,7 +114,7 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q, > |???????? xe_exec_queue_get(job->q); > | |???????? err = drm_sched_job_init(&job->drm, q->entity, 1, NULL, > | -??????????????????????????????? q->xef->drm->client_id); > | +??????????????????????????????? q->xef ? q->xef->drm->client_id : 0); > |???????? if (err) > |???????????????? goto err_free; > > I tested with the above diff and it at least loads... Thanks for looking into this, the change looks fine to me. Pierre-Eric > > Also, I see this in intel-xe mailing list, but I'm not sure why we > didn't have any CI results... I will check that. > > Lucas De Marchi From lucas.demarchi at intel.com Wed May 28 19:07:34 2025 From: lucas.demarchi at intel.com (Lucas De Marchi) Date: Wed, 28 May 2025 14:07:34 -0500 Subject: [PATCH v11 02/10] drm/sched: Store the drm client_id in drm_sched_fence In-Reply-To: <20250526125505.2360-3-pierre-eric.pelloux-prayer@amd.com> References: <20250526125505.2360-1-pierre-eric.pelloux-prayer@amd.com> <20250526125505.2360-3-pierre-eric.pelloux-prayer@amd.com> Message-ID: <5jnpsmjef5ibegbsbelkfmudv4wagpcfb25nptqs5z4ccitq4c@3bdtrbrrmtil> On Mon, May 26, 2025 at 02:54:44PM +0200, Pierre-Eric Pelloux-Prayer wrote: > drivers/gpu/drm/xe/xe_sched_job.c | 3 ++- >diff --git a/drivers/gpu/drm/xe/xe_sched_job.c b/drivers/gpu/drm/xe/xe_sched_job.c >index f0a6ce610948..5921293b25db 100644 >--- a/drivers/gpu/drm/xe/xe_sched_job.c >+++ b/drivers/gpu/drm/xe/xe_sched_job.c >@@ -113,7 +113,8 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q, > kref_init(&job->refcount); > xe_exec_queue_get(job->q); > >- err = drm_sched_job_init(&job->drm, q->entity, 1, NULL); >+ err = drm_sched_job_init(&job->drm, q->entity, 1, NULL, >+ q->xef->drm->client_id); you can't do this here. xef is only !NULL if it's a job from userspace. For in-kernel jobs, xef is NULL and this explodes. Right now this completely breaks xe since one of the very first things we do is to submit a job to save the default context. Example: https://intel-gfx-ci.01.org/tree/intel-xe/xe-3151-56d2b14961751a677ff1f7ff8b93a6c814ce2be3/bat-bmg-1/igt at xe_module_load@load.html <4> [] RIP: 0010:xe_sched_job_create+0xbd/0x390 [xe] <4> [] Code: c1 43 18 85 c0 0f 84 6f 02 00 00 8d 50 01 09 c2 0f 88 3e 02 00 00 48 8b 03 48 8b b3 d8 00 00 00 31 c9 4c 89 ef ba 01 00 00 00 <48> 8b 40 08 4c 8b 40 60 e8 86 64 7c ff 41 89 c4 85 c0 0f 85 9b 01 <4> [] RSP: 0018:ffffc900031972d8 EFLAGS: 00010246 <4> [] RAX: 0000000000000000 RBX: ffff88815fc40d00 RCX: 0000000000000000 <4> [] RDX: 0000000000000001 RSI: ffff88812e6552a8 RDI: ffff88815f939c40 <4> [] RBP: ffffc90003197318 R08: 0000000000000000 R09: 0000000000000000 <4> [] R10: 0000000000000000 R11: 0000000000000000 R12: ffffc90003197428 <4> [] R13: ffff88815f939c40 R14: ffff88811f054000 R15: ffff88815fc40d00 <4> [] FS: 00007681f2948940(0000) GS:ffff8888daf14000(0000) knlGS:0000000000000000 <4> [] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4> [] CR2: 0000000000000008 CR3: 0000000118315004 CR4: 0000000000f72ef0 <4> [] PKRU: 55555554 <4> [] Call Trace: <4> [] <4> [] __xe_bb_create_job+0xa2/0x240 [xe] <4> [] ? find_held_lock+0x31/0x90 <4> [] ? xa_find_after+0x12c/0x250 <4> [] xe_bb_create_job+0x6e/0x380 [xe] <4> [] ? xa_find_after+0x136/0x250 <4> [] ? __drm_dev_dbg+0x7d/0xb0 <4> [] xe_gt_record_default_lrcs+0x542/0xb00 [xe] Can we use 0 for in-kernel client since drm_file starts them from 1? Like this: | diff --git a/drivers/gpu/drm/xe/xe_sched_job.c b/drivers/gpu/drm/xe/xe_sched_job.c | index 5921293b25db3..d21bf8f269640 100644 | --- a/drivers/gpu/drm/xe/xe_sched_job.c | +++ b/drivers/gpu/drm/xe/xe_sched_job.c | @@ -114,7 +114,7 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q, | xe_exec_queue_get(job->q); | | err = drm_sched_job_init(&job->drm, q->entity, 1, NULL, | - q->xef->drm->client_id); | + q->xef ? q->xef->drm->client_id : 0); | if (err) | goto err_free; I tested with the above diff and it at least loads... Also, I see this in intel-xe mailing list, but I'm not sure why we didn't have any CI results... I will check that. Lucas De Marchi From mark.filion at collabora.com Thu May 29 19:01:39 2025 From: mark.filion at collabora.com (Mark Filion) Date: Thu, 29 May 2025 15:01:39 -0400 Subject: XDC 2025: Registration & Call for Proposals now open! Message-ID: Hello! Registration & Call for Proposals are now open for XDC 2025, which will take place at the TU Wien Kuppelsaal in Vienna, Austria on 29 September to 1 October. https://xdc2025.x.org As usual, the conference is free of charge and open to the general public. If you plan on attending, please make sure to register as early as possible: https://indico.freedesktop.org/event/10/registrations/ In addition to registration, the CfP is now open for talks, demos, and workshops at XDC 2025. While any serious proposal will be gratefully considered, topics of interest to X.Org and freedesktop.org developers are encouraged. The program focus is on new development, ongoing challenges and anything else that will spark discussions among attendees in the hallway track. We are open to talks across all layers of the graphics stack, from the kernel to desktop environments / graphical applications and about how to make things better for the developers who build them. Head to the CfP page to learn more: https://indico.freedesktop.org/event/10/abstracts/ The deadline for submissions Friday, 11 July 2025. We are looking forward to seeing you in Vienna! If you have any questions, please email the organizer (hfink at snap.com), adding on CC the X.org board (board at foundation.x.org). And don't forget, you can follow us on Mastodon for all the latest updates and to stay connected: https://floss.social/@XOrgDevConf Best, Mark From mcanal at igalia.com Fri May 30 14:01:30 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Fri, 30 May 2025 11:01:30 -0300 Subject: [PATCH v2 1/8] drm/sched: Rename DRM_GPU_SCHED_STAT_NOMINAL to DRM_GPU_SCHED_STAT_RESET In-Reply-To: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> References: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> Message-ID: <20250530-sched-skip-reset-v2-1-c40a8d2d8daa@igalia.com> Among the scheduler's statuses, the only one that indicates an error is DRM_GPU_SCHED_STAT_ENODEV. Any status other than DRM_GPU_SCHED_STAT_ENODEV signifies that the operation succeeded and the GPU is in a nominal state. However, to provide more information about the GPU's status, it is needed to convey more information than just "OK". Therefore, rename DRM_GPU_SCHED_STAT_NOMINAL to DRM_GPU_SCHED_STAT_RESET, which betters communicates the meaning of this status. The status DRM_GPU_SCHED_STAT_RESET indicates that the GPU has hung, but it has been successfully reset and is now in a nominal state again. Signed-off-by: Ma?ra Canal --- To: Min Ma To: Lizhi Hou To: Oded Gabbay To: Frank Binns To: Matt Coster To: Qiang Yu To: Lyude Paul To: Alex Deucher To: Christian K?nig --- drivers/accel/amdxdna/aie2_ctx.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 2 +- drivers/gpu/drm/etnaviv/etnaviv_sched.c | 4 ++-- drivers/gpu/drm/imagination/pvr_queue.c | 4 ++-- drivers/gpu/drm/lima/lima_sched.c | 6 +++--- drivers/gpu/drm/nouveau/nouveau_exec.c | 2 +- drivers/gpu/drm/nouveau/nouveau_sched.c | 2 +- drivers/gpu/drm/panfrost/panfrost_job.c | 6 +++--- drivers/gpu/drm/panthor/panthor_mmu.c | 2 +- drivers/gpu/drm/panthor/panthor_sched.c | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 2 +- drivers/gpu/drm/scheduler/tests/mock_scheduler.c | 2 +- drivers/gpu/drm/v3d/v3d_sched.c | 6 +++--- drivers/gpu/drm/xe/xe_guc_submit.c | 6 +++--- include/drm/gpu_scheduler.h | 4 ++-- 15 files changed, 26 insertions(+), 26 deletions(-) diff --git a/drivers/accel/amdxdna/aie2_ctx.c b/drivers/accel/amdxdna/aie2_ctx.c index 3e38a5f637ea049c9b8f08dc270482377a4fc4f3..756f817cc8fe523041ab11a6454e0f42d9781f9e 100644 --- a/drivers/accel/amdxdna/aie2_ctx.c +++ b/drivers/accel/amdxdna/aie2_ctx.c @@ -361,7 +361,7 @@ aie2_sched_job_timedout(struct drm_sched_job *sched_job) aie2_hwctx_restart(xdna, hwctx); mutex_unlock(&xdna->dev_lock); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } static const struct drm_sched_backend_ops sched_ops = { diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c index 75262ce8db27fad4a1f1af00639031b040f21c87..863e1158196cc0db0296abadd2302de1c700649b 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c @@ -199,7 +199,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job) exit: drm_dev_exit(idx); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm *vm, diff --git a/drivers/gpu/drm/etnaviv/etnaviv_sched.c b/drivers/gpu/drm/etnaviv/etnaviv_sched.c index 76a3a3e517d8d9f654fb6b9e98e72910795cfc7a..7146069a98492f5fab2a49d96e2054f649e1fe3d 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_sched.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_sched.c @@ -86,11 +86,11 @@ static enum drm_gpu_sched_stat etnaviv_sched_timedout_job(struct drm_sched_job drm_sched_resubmit_jobs(&gpu->sched); drm_sched_start(&gpu->sched, 0); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; out_no_timeout: list_add(&sched_job->list, &sched_job->sched->pending_list); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } static void etnaviv_sched_free_job(struct drm_sched_job *sched_job) diff --git a/drivers/gpu/drm/imagination/pvr_queue.c b/drivers/gpu/drm/imagination/pvr_queue.c index 5a41ee79fed646a86344cd16e78efdb45ff02e43..fc415dd0d7a73631bd4144c9f35b9b294c625a12 100644 --- a/drivers/gpu/drm/imagination/pvr_queue.c +++ b/drivers/gpu/drm/imagination/pvr_queue.c @@ -803,7 +803,7 @@ static void pvr_queue_start(struct pvr_queue *queue) * the scheduler, and re-assign parent fences in the middle. * * Return: - * * DRM_GPU_SCHED_STAT_NOMINAL. + * * DRM_GPU_SCHED_STAT_RESET. */ static enum drm_gpu_sched_stat pvr_queue_timedout_job(struct drm_sched_job *s_job) @@ -854,7 +854,7 @@ pvr_queue_timedout_job(struct drm_sched_job *s_job) drm_sched_start(sched, 0); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } /** diff --git a/drivers/gpu/drm/lima/lima_sched.c b/drivers/gpu/drm/lima/lima_sched.c index 954f4325b859b2977a2cc608a99a6ebb642f1000..739e8c6c6d909aa4263bad8a12ec07f0c6607bb2 100644 --- a/drivers/gpu/drm/lima/lima_sched.c +++ b/drivers/gpu/drm/lima/lima_sched.c @@ -412,7 +412,7 @@ static enum drm_gpu_sched_stat lima_sched_timedout_job(struct drm_sched_job *job */ if (dma_fence_is_signaled(task->fence)) { DRM_WARN("%s spurious timeout\n", lima_ip_name(ip)); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } /* @@ -429,7 +429,7 @@ static enum drm_gpu_sched_stat lima_sched_timedout_job(struct drm_sched_job *job if (dma_fence_is_signaled(task->fence)) { DRM_WARN("%s unexpectedly high interrupt latency\n", lima_ip_name(ip)); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } /* @@ -467,7 +467,7 @@ static enum drm_gpu_sched_stat lima_sched_timedout_job(struct drm_sched_job *job drm_sched_resubmit_jobs(&pipe->base); drm_sched_start(&pipe->base, 0); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } static void lima_sched_free_job(struct drm_sched_job *job) diff --git a/drivers/gpu/drm/nouveau/nouveau_exec.c b/drivers/gpu/drm/nouveau/nouveau_exec.c index a0b5f1b16e8b39a84e3e79429a532b10fbffd066..1b89f1b36b89c7ea4f4ca2f3ea799511858fc068 100644 --- a/drivers/gpu/drm/nouveau/nouveau_exec.c +++ b/drivers/gpu/drm/nouveau/nouveau_exec.c @@ -185,7 +185,7 @@ nouveau_exec_job_timeout(struct nouveau_job *job) NV_PRINTK(warn, job->cli, "job timeout, channel %d killed!\n", chan->chid); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } static const struct nouveau_job_ops nouveau_exec_job_ops = { diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.c b/drivers/gpu/drm/nouveau/nouveau_sched.c index 460a5fb024129a0557f2b55008278e1378019d89..e60f7892f5ce9aff0c5fa1908c1a0445891927ed 100644 --- a/drivers/gpu/drm/nouveau/nouveau_sched.c +++ b/drivers/gpu/drm/nouveau/nouveau_sched.c @@ -371,7 +371,7 @@ nouveau_sched_timedout_job(struct drm_sched_job *sched_job) { struct drm_gpu_scheduler *sched = sched_job->sched; struct nouveau_job *job = to_nouveau_job(sched_job); - enum drm_gpu_sched_stat stat = DRM_GPU_SCHED_STAT_NOMINAL; + enum drm_gpu_sched_stat stat = DRM_GPU_SCHED_STAT_RESET; drm_sched_stop(sched, sched_job); diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c index 5657106c2f7d0a0ca6162850767f58f3200cce13..afcffe7f8fe9e11f84e4ab7e8f5a72f7bf583690 100644 --- a/drivers/gpu/drm/panfrost/panfrost_job.c +++ b/drivers/gpu/drm/panfrost/panfrost_job.c @@ -755,7 +755,7 @@ static enum drm_gpu_sched_stat panfrost_job_timedout(struct drm_sched_job * spurious. Bail out. */ if (dma_fence_is_signaled(job->done_fence)) - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; /* * Panfrost IRQ handler may take a long time to process an interrupt @@ -770,7 +770,7 @@ static enum drm_gpu_sched_stat panfrost_job_timedout(struct drm_sched_job if (dma_fence_is_signaled(job->done_fence)) { dev_warn(pfdev->dev, "unexpectedly high interrupt latency\n"); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } dev_err(pfdev->dev, "gpu sched timeout, js=%d, config=0x%x, status=0x%x, head=0x%x, tail=0x%x, sched_job=%p", @@ -786,7 +786,7 @@ static enum drm_gpu_sched_stat panfrost_job_timedout(struct drm_sched_job atomic_set(&pfdev->reset.pending, 1); panfrost_reset(pfdev, sched_job); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } static void panfrost_reset_work(struct work_struct *work) diff --git a/drivers/gpu/drm/panthor/panthor_mmu.c b/drivers/gpu/drm/panthor/panthor_mmu.c index 4ae72b2117937f609593e47686d944854111edd7..96bad928a7ceb3993aefb423a792145c8aa9fb92 100644 --- a/drivers/gpu/drm/panthor/panthor_mmu.c +++ b/drivers/gpu/drm/panthor/panthor_mmu.c @@ -2282,7 +2282,7 @@ static enum drm_gpu_sched_stat panthor_vm_bind_timedout_job(struct drm_sched_job *sched_job) { WARN(1, "VM_BIND ops are synchronous for now, there should be no timeout!"); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } static const struct drm_sched_backend_ops panthor_vm_bind_ops = { diff --git a/drivers/gpu/drm/panthor/panthor_sched.c b/drivers/gpu/drm/panthor/panthor_sched.c index a2248f692a030c1c84869b9a1948ad1cb0c0b490..8f17394cc82aad9eaf01e473cd9d3dea46fa3d61 100644 --- a/drivers/gpu/drm/panthor/panthor_sched.c +++ b/drivers/gpu/drm/panthor/panthor_sched.c @@ -3241,7 +3241,7 @@ queue_timedout_job(struct drm_sched_job *sched_job) queue_start(queue); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } static void queue_free_job(struct drm_sched_job *sched_job) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index d20726d7adf0212a3651a3f64544f308e7412b4a..3b0760dfa4fe2fc63e893cda733e78d08dd451d5 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -540,7 +540,7 @@ static void drm_sched_job_timedout(struct work_struct *work) { struct drm_gpu_scheduler *sched; struct drm_sched_job *job; - enum drm_gpu_sched_stat status = DRM_GPU_SCHED_STAT_NOMINAL; + enum drm_gpu_sched_stat status = DRM_GPU_SCHED_STAT_RESET; sched = container_of(work, struct drm_gpu_scheduler, work_tdr.work); diff --git a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c index fcddaeaa921760715a344b81ebbed6cc921231ac..fdf5f34b39e02c8a8648d8bea566a27fd3251516 100644 --- a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c +++ b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c @@ -208,7 +208,7 @@ mock_sched_timedout_job(struct drm_sched_job *sched_job) job->flags |= DRM_MOCK_SCHED_JOB_TIMEDOUT; - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } static void mock_sched_free_job(struct drm_sched_job *sched_job) diff --git a/drivers/gpu/drm/v3d/v3d_sched.c b/drivers/gpu/drm/v3d/v3d_sched.c index 466d28ceee28d6c173197b07c0d5f73c8db53e2c..e1997387831541fb053e472672004cf511c25558 100644 --- a/drivers/gpu/drm/v3d/v3d_sched.c +++ b/drivers/gpu/drm/v3d/v3d_sched.c @@ -741,7 +741,7 @@ v3d_gpu_reset_for_timeout(struct v3d_dev *v3d, struct drm_sched_job *sched_job) mutex_unlock(&v3d->reset_lock); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } /* If the current address or return address have changed, then the GPU @@ -761,7 +761,7 @@ v3d_cl_job_timedout(struct drm_sched_job *sched_job, enum v3d_queue q, if (*timedout_ctca != ctca || *timedout_ctra != ctra) { *timedout_ctca = ctca; *timedout_ctra = ctra; - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } return v3d_gpu_reset_for_timeout(v3d, sched_job); @@ -805,7 +805,7 @@ v3d_csd_job_timedout(struct drm_sched_job *sched_job) */ if (job->timedout_batches != batches) { job->timedout_batches = batches; - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } return v3d_gpu_reset_for_timeout(v3d, sched_job); diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c index 813c3c0bb2500722b03831d9815e83400460c9e2..98363d688cbbf884e17e6610366202a3372f5fe0 100644 --- a/drivers/gpu/drm/xe/xe_guc_submit.c +++ b/drivers/gpu/drm/xe/xe_guc_submit.c @@ -1071,7 +1071,7 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job) xe_sched_add_pending_job(sched, job); xe_sched_submission_start(sched); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } /* Kill the run_job entry point */ @@ -1237,7 +1237,7 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job) /* Start fence signaling */ xe_hw_fence_irq_start(q->fence_irq); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; sched_enable: enable_scheduling(q); @@ -1250,7 +1250,7 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job) xe_sched_add_pending_job(sched, job); xe_sched_submission_start(sched); - return DRM_GPU_SCHED_STAT_NOMINAL; + return DRM_GPU_SCHED_STAT_RESET; } static void __guc_exec_queue_fini_async(struct work_struct *w) diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index e62a7214e05217d72de5c6e5168544d47099090a..83e5c00d8dd9a83ab20547a93d6fc572de97616e 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -391,12 +391,12 @@ struct drm_sched_job { * enum drm_gpu_sched_stat - the scheduler's status * * @DRM_GPU_SCHED_STAT_NONE: Reserved. Do not use. - * @DRM_GPU_SCHED_STAT_NOMINAL: Operation succeeded. + * @DRM_GPU_SCHED_STAT_RESET: The GPU hung and successfully reset. * @DRM_GPU_SCHED_STAT_ENODEV: Error: Device is not available anymore. */ enum drm_gpu_sched_stat { DRM_GPU_SCHED_STAT_NONE, - DRM_GPU_SCHED_STAT_NOMINAL, + DRM_GPU_SCHED_STAT_RESET, DRM_GPU_SCHED_STAT_ENODEV, }; -- 2.49.0 From mcanal at igalia.com Fri May 30 14:01:29 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Fri, 30 May 2025 11:01:29 -0300 Subject: [PATCH v2 0/8] drm/sched: Allow drivers to skip the reset with DRM_GPU_SCHED_STAT_NO_HANG Message-ID: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> When the DRM scheduler times out, it's possible that the GPU isn't hung; instead, a job may still be running, and there may be no valid reason to reset the hardware. This can occur in two situations: 1. The GPU exposes some mechanism that ensures the GPU is still making progress. By checking this mechanism, the driver can safely skip the reset, re-arm the timeout, and allow the job to continue running until completion. This is the case for v3d, Etnaviv, and Xe. 2. Timeout has fired before the free-job worker. Consequently, the scheduler calls `timedout_job()` for a job that isn't timed out. These two scenarios are problematic because the job was removed from the `sched->pending_list` before calling `sched->ops->timedout_job()`, which means that when the job finishes, it won't be freed by the scheduler though `sched->ops->free_job()`. As a result, the job and its resources won't be freed, leading to a memory leak. For v3d specifically, we have observed that these memory leaks can be significant in certain scenarios, as reported by users in [1][2]. To address this situation, I submitted a patch similar to commit 704d3d60fec4 ("drm/etnaviv: don't block scheduler when GPU is still active") for v3d [3]. This patch has already landed in drm-misc-fixes and successfully resolved the users' issues. However, as I mentioned in [3], exposing the scheduler's internals within the drivers isn't ideal and I believe this specific situation can be addressed within the DRM scheduler framework. This series aims to resolve this issue by adding a new DRM sched status that allows a driver to skip the reset. This new status will indicate that the job should be reinserted into the pending list, and the driver will still signal its completion. [1] https://gitlab.freedesktop.org/mesa/mesa/-/issues/12227 [2] https://github.com/raspberrypi/linux/issues/6817 [3] https://lore.kernel.org/dri-devel/20250430210643.57924-1-mcanal at igalia.com/T/ Best Regards, - Ma?ra --- v1 -> v2: - Fix several grammar nits across the documentation and commit messages. - Drop "drm/sched: Always free the job after the timeout" (Tvrtko Ursulin) - [1/8] NEW PATCH: Rename DRM_GPU_SCHED_STAT_NOMINAL to a more semantic name (Tvrtko Ursulin, Philipp Stanner) - [2/8] Rename DRM_GPU_SCHED_STAT_RUNNING to DRM_GPU_SCHED_STAT_NO_HANG (Tvrtko Ursulin, Philipp Stanner) - [2/8] Requeue free-job work after reinserting the job to the pending list (Matthew Brost) - [2/8] Create a helper function to reinsert the job (Philipp Stanner) - [2/8] Rewrite the commit message (Philipp Stanner) - [2/8] Add a comment to `drm_sched_start()` documentation, similar to what was commented in `drm_sched_stop()` (Philipp Stanner) - [3/8] Keep HZ as timeout for `drm_mock_sched_job_wait_scheduled()` (Tvrtko Ursulin) - [4/8] Use a job flag to indicate that `timedout_job()` should skip the reset (Tvrtko Ursulin) - [7/8] Use DRM_GPU_SCHED_STAT_NO_HANG to re-arm the timer in other cases as well (Matthew Brost) - Link to v1: https://lore.kernel.org/r/20250503-sched-skip-reset-v1-0-ed0d6701a3fe at igalia.com --- Ma?ra Canal (8): drm/sched: Rename DRM_GPU_SCHED_STAT_NOMINAL to DRM_GPU_SCHED_STAT_RESET drm/sched: Allow drivers to skip the reset and keep on running drm/sched: Reduce scheduler's timeout for timeout tests drm/sched: Add new test for DRM_GPU_SCHED_STAT_NO_HANG drm/v3d: Use DRM_GPU_SCHED_STAT_NO_HANG to skip the reset drm/etnaviv: Use DRM_GPU_SCHED_STAT_NO_HANG to skip the reset drm/xe: Use DRM_GPU_SCHED_STAT_NO_HANG to skip the reset drm/panfrost: Use DRM_GPU_SCHED_STAT_NO_HANG to skip the reset drivers/accel/amdxdna/aie2_ctx.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 2 +- drivers/gpu/drm/etnaviv/etnaviv_sched.c | 13 +++--- drivers/gpu/drm/imagination/pvr_queue.c | 4 +- drivers/gpu/drm/lima/lima_sched.c | 6 +-- drivers/gpu/drm/nouveau/nouveau_exec.c | 2 +- drivers/gpu/drm/nouveau/nouveau_sched.c | 2 +- drivers/gpu/drm/panfrost/panfrost_job.c | 10 ++--- drivers/gpu/drm/panthor/panthor_mmu.c | 2 +- drivers/gpu/drm/panthor/panthor_sched.c | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 51 ++++++++++++++++++++++-- drivers/gpu/drm/scheduler/tests/mock_scheduler.c | 7 +++- drivers/gpu/drm/scheduler/tests/sched_tests.h | 1 + drivers/gpu/drm/scheduler/tests/tests_basic.c | 51 ++++++++++++++++++++++-- drivers/gpu/drm/v3d/v3d_sched.c | 6 +-- drivers/gpu/drm/xe/xe_guc_submit.c | 14 ++----- include/drm/gpu_scheduler.h | 7 +++- 17 files changed, 136 insertions(+), 46 deletions(-) --- base-commit: df1c3093aee3daee4d56a5cb8cdba75c1ef6962f change-id: 20250502-sched-skip-reset-bf7c163233da From mcanal at igalia.com Fri May 30 14:01:31 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Fri, 30 May 2025 11:01:31 -0300 Subject: [PATCH v2 2/8] drm/sched: Allow drivers to skip the reset and keep on running In-Reply-To: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> References: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> Message-ID: <20250530-sched-skip-reset-v2-2-c40a8d2d8daa@igalia.com> When the DRM scheduler times out, it's possible that the GPU isn't hung; instead, a job may still be running, and there may be no valid reason to reset the hardware. This can occur in two situations: 1. The GPU exposes some mechanism that ensures the GPU is still making progress. By checking this mechanism, the driver can safely skip the reset, re-arm the timeout, and allow the job to continue running until completion. This is the case for v3d, Etnaviv, and Xe. 2. Timeout has fired before the free-job worker. Consequently, the scheduler calls `timedout_job()` for a job that isn't timed out. These two scenarios are problematic because the job was removed from the `sched->pending_list` before calling `sched->ops->timedout_job()`, which means that when the job finishes, it won't be freed by the scheduler though `sched->ops->free_job()`. As a result, the job and its resources won't be freed, leading to a memory leak. To resolve those scenarios, create a new `drm_gpu_sched_stat`, called DRM_GPU_SCHED_STAT_NO_HANG, that allows a driver to skip the reset. The new status will indicate that the job should be reinserted into the pending list, and the hardware / driver is still responsible to signal job completion. Signed-off-by: Ma?ra Canal --- drivers/gpu/drm/scheduler/sched_main.c | 49 ++++++++++++++++++++++++++++++++-- include/drm/gpu_scheduler.h | 3 +++ 2 files changed, 50 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 3b0760dfa4fe2fc63e893cda733e78d08dd451d5..ddc53eadab7bb6a15109f43989afa1f7a95a3b41 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -379,11 +379,16 @@ static void drm_sched_run_free_queue(struct drm_gpu_scheduler *sched) { struct drm_sched_job *job; - spin_lock(&sched->job_list_lock); job = list_first_entry_or_null(&sched->pending_list, struct drm_sched_job, list); if (job && dma_fence_is_signaled(&job->s_fence->finished)) __drm_sched_run_free_queue(sched); +} + +static void drm_sched_run_free_queue_unlocked(struct drm_gpu_scheduler *sched) +{ + spin_lock(&sched->job_list_lock); + drm_sched_run_free_queue(sched); spin_unlock(&sched->job_list_lock); } @@ -536,6 +541,32 @@ static void drm_sched_job_begin(struct drm_sched_job *s_job) spin_unlock(&sched->job_list_lock); } +/** + * drm_sched_job_reinsert_on_false_timeout - Reinsert the job on a false timeout + * + * @sched: scheduler instance + * @job: job to be reinserted on the pending list + * + * In the case of a "false timeout" - when a timeout occurs but the GPU isn't + * hung and the job is making progress, the scheduler must reinsert the job back + * into the pending list. Otherwise, the job and its resources won't be freed + * through the &drm_sched_backend_ops.free_job callback. + * + * Note that after reinserting the job, the scheduler enqueues the free-job + * work again if ready. Otherwise, a signaled job could be added to the pending + * list, but never freed. + * + * This function must be used in "false timeout" cases only. + */ +static void drm_sched_job_reinsert_on_false_timeout(struct drm_gpu_scheduler *sched, + struct drm_sched_job *job) +{ + spin_lock(&sched->job_list_lock); + list_add(&job->list, &sched->pending_list); + drm_sched_run_free_queue(sched); + spin_unlock(&sched->job_list_lock); +} + static void drm_sched_job_timedout(struct work_struct *work) { struct drm_gpu_scheduler *sched; @@ -569,6 +600,14 @@ static void drm_sched_job_timedout(struct work_struct *work) job->sched->ops->free_job(job); sched->free_guilty = false; } + + /* + * If the driver indicated that the GPU is still running and wants + * to skip the reset, reinsert the job back into the pending list + * and re-arm the timeout. + */ + if (status == DRM_GPU_SCHED_STAT_NO_HANG) + drm_sched_job_reinsert_on_false_timeout(sched, job); } else { spin_unlock(&sched->job_list_lock); } @@ -591,6 +630,9 @@ static void drm_sched_job_timedout(struct work_struct *work) * This function is typically used for reset recovery (see the docu of * drm_sched_backend_ops.timedout_job() for details). Do not call it for * scheduler teardown, i.e., before calling drm_sched_fini(). + * + * As it's used for reset recovery, drm_sched_stop() shouldn't be called + * if the driver skipped the timeout (DRM_GPU_SCHED_STAT_NO_HANG). */ void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad) { @@ -676,6 +718,9 @@ EXPORT_SYMBOL(drm_sched_stop); * drm_sched_backend_ops.timedout_job() for details). Do not call it for * scheduler startup. The scheduler itself is fully operational after * drm_sched_init() succeeded. + * + * As it's used for reset recovery, drm_sched_start() shouldn't be called + * if the driver skipped the timeout (DRM_GPU_SCHED_STAT_NO_HANG). */ void drm_sched_start(struct drm_gpu_scheduler *sched, int errno) { @@ -1197,7 +1242,7 @@ static void drm_sched_free_job_work(struct work_struct *w) if (job) sched->ops->free_job(job); - drm_sched_run_free_queue(sched); + drm_sched_run_free_queue_unlocked(sched); drm_sched_run_job_queue(sched); } diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 83e5c00d8dd9a83ab20547a93d6fc572de97616e..063c1915841aa54a0859bdccd3c1ef6028105bec 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -393,11 +393,14 @@ struct drm_sched_job { * @DRM_GPU_SCHED_STAT_NONE: Reserved. Do not use. * @DRM_GPU_SCHED_STAT_RESET: The GPU hung and successfully reset. * @DRM_GPU_SCHED_STAT_ENODEV: Error: Device is not available anymore. + * @DRM_GPU_SCHED_STAT_NO_HANG: Contrary to scheduler's belief, the GPU + * did not hang and it's operational. */ enum drm_gpu_sched_stat { DRM_GPU_SCHED_STAT_NONE, DRM_GPU_SCHED_STAT_RESET, DRM_GPU_SCHED_STAT_ENODEV, + DRM_GPU_SCHED_STAT_NO_HANG, }; /** -- 2.49.0 From mcanal at igalia.com Fri May 30 14:01:32 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Fri, 30 May 2025 11:01:32 -0300 Subject: [PATCH v2 3/8] drm/sched: Reduce scheduler's timeout for timeout tests In-Reply-To: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> References: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> Message-ID: <20250530-sched-skip-reset-v2-3-c40a8d2d8daa@igalia.com> As more KUnit tests are introduced to evaluate the basic capabilities of the `timedout_job()` hook, the test suite will continue to increase in duration. To reduce the overall running time of the test suite, decrease the scheduler's timeout for the timeout tests. Before this commit: [15:42:26] Elapsed time: 15.637s total, 0.002s configuring, 10.387s building, 5.229s running After this commit: [15:45:26] Elapsed time: 9.263s total, 0.002s configuring, 5.168s building, 4.037s running Signed-off-by: Ma?ra Canal --- drivers/gpu/drm/scheduler/tests/tests_basic.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/scheduler/tests/tests_basic.c b/drivers/gpu/drm/scheduler/tests/tests_basic.c index 7230057e0594c6246f02608f07fcb1f8d738ac75..41c648782f4548e202bd8711b45d28eead9bd0b2 100644 --- a/drivers/gpu/drm/scheduler/tests/tests_basic.c +++ b/drivers/gpu/drm/scheduler/tests/tests_basic.c @@ -5,6 +5,8 @@ #include "sched_tests.h" +#define MOCK_TIMEOUT (HZ / 5) + /* * DRM scheduler basic tests should check the basic functional correctness of * the scheduler, including some very light smoke testing. More targeted tests, @@ -28,7 +30,7 @@ static void drm_sched_basic_exit(struct kunit *test) static int drm_sched_timeout_init(struct kunit *test) { - test->priv = drm_mock_sched_new(test, HZ); + test->priv = drm_mock_sched_new(test, MOCK_TIMEOUT); return 0; } @@ -227,14 +229,14 @@ static void drm_sched_basic_timeout(struct kunit *test) done = drm_mock_sched_job_wait_scheduled(job, HZ); KUNIT_ASSERT_TRUE(test, done); - done = drm_mock_sched_job_wait_finished(job, HZ / 2); + done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT / 2); KUNIT_ASSERT_FALSE(test, done); KUNIT_ASSERT_EQ(test, job->flags & DRM_MOCK_SCHED_JOB_TIMEDOUT, 0); - done = drm_mock_sched_job_wait_finished(job, HZ); + done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT); KUNIT_ASSERT_FALSE(test, done); KUNIT_ASSERT_EQ(test, -- 2.49.0 From mcanal at igalia.com Fri May 30 14:01:33 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Fri, 30 May 2025 11:01:33 -0300 Subject: [PATCH v2 4/8] drm/sched: Add new test for DRM_GPU_SCHED_STAT_NO_HANG In-Reply-To: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> References: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> Message-ID: <20250530-sched-skip-reset-v2-4-c40a8d2d8daa@igalia.com> Add a test to submit a single job against a scheduler with the timeout configured and verify that if the job is still running, the timeout handler will skip the reset and allow the job to complete. Signed-off-by: Ma?ra Canal --- drivers/gpu/drm/scheduler/tests/mock_scheduler.c | 5 +++ drivers/gpu/drm/scheduler/tests/sched_tests.h | 1 + drivers/gpu/drm/scheduler/tests/tests_basic.c | 43 ++++++++++++++++++++++++ 3 files changed, 49 insertions(+) diff --git a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c index fdf5f34b39e02c8a8648d8bea566a27fd3251516..39429f5cd19ee3c23816f257d566b47d3daa4baa 100644 --- a/drivers/gpu/drm/scheduler/tests/mock_scheduler.c +++ b/drivers/gpu/drm/scheduler/tests/mock_scheduler.c @@ -208,6 +208,11 @@ mock_sched_timedout_job(struct drm_sched_job *sched_job) job->flags |= DRM_MOCK_SCHED_JOB_TIMEDOUT; + if (job->flags & DRM_MOCK_SCHED_JOB_DONT_RESET) { + job->flags &= ~DRM_MOCK_SCHED_JOB_DONT_RESET; + return DRM_GPU_SCHED_STAT_NO_HANG; + } + return DRM_GPU_SCHED_STAT_RESET; } diff --git a/drivers/gpu/drm/scheduler/tests/sched_tests.h b/drivers/gpu/drm/scheduler/tests/sched_tests.h index 27caf8285fb74b9f3c9ce2daa1c44d4a0c967e92..5259f181e55387c41efbcd3f6addc9465331d787 100644 --- a/drivers/gpu/drm/scheduler/tests/sched_tests.h +++ b/drivers/gpu/drm/scheduler/tests/sched_tests.h @@ -98,6 +98,7 @@ struct drm_mock_sched_job { #define DRM_MOCK_SCHED_JOB_DONE 0x1 #define DRM_MOCK_SCHED_JOB_TIMEDOUT 0x2 +#define DRM_MOCK_SCHED_JOB_DONT_RESET 0x4 unsigned long flags; struct list_head link; diff --git a/drivers/gpu/drm/scheduler/tests/tests_basic.c b/drivers/gpu/drm/scheduler/tests/tests_basic.c index 41c648782f4548e202bd8711b45d28eead9bd0b2..2ba2d1b0c3cad9626ab9d89cfae05244c670a826 100644 --- a/drivers/gpu/drm/scheduler/tests/tests_basic.c +++ b/drivers/gpu/drm/scheduler/tests/tests_basic.c @@ -246,8 +246,51 @@ static void drm_sched_basic_timeout(struct kunit *test) drm_mock_sched_entity_free(entity); } +static void drm_sched_skip_reset(struct kunit *test) +{ + struct drm_mock_scheduler *sched = test->priv; + struct drm_mock_sched_entity *entity; + struct drm_mock_sched_job *job; + bool done; + + /* + * Submit a single job against a scheduler with the timeout configured + * and verify that if the job is still running, the timeout handler + * will skip the reset and allow the job to complete. + */ + + entity = drm_mock_sched_entity_new(test, + DRM_SCHED_PRIORITY_NORMAL, + sched); + job = drm_mock_sched_job_new(test, entity); + + job->flags = DRM_MOCK_SCHED_JOB_DONT_RESET; + + drm_mock_sched_job_set_duration_us(job, jiffies_to_usecs(2 * MOCK_TIMEOUT)); + drm_mock_sched_job_submit(job); + + done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT); + KUNIT_ASSERT_FALSE(test, done); + + KUNIT_ASSERT_EQ(test, + job->flags & DRM_MOCK_SCHED_JOB_TIMEDOUT, + DRM_MOCK_SCHED_JOB_TIMEDOUT); + + KUNIT_ASSERT_EQ(test, + job->flags & DRM_MOCK_SCHED_JOB_DONT_RESET, + 0); + + KUNIT_ASSERT_FALSE(test, list_empty(&sched->job_list)); + + done = drm_mock_sched_job_wait_finished(job, MOCK_TIMEOUT); + KUNIT_ASSERT_TRUE(test, done); + + drm_mock_sched_entity_free(entity); +} + static struct kunit_case drm_sched_timeout_tests[] = { KUNIT_CASE(drm_sched_basic_timeout), + KUNIT_CASE(drm_sched_skip_reset), {} }; -- 2.49.0 From mcanal at igalia.com Fri May 30 14:01:34 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Fri, 30 May 2025 11:01:34 -0300 Subject: [PATCH v2 5/8] drm/v3d: Use DRM_GPU_SCHED_STAT_NO_HANG to skip the reset In-Reply-To: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> References: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> Message-ID: <20250530-sched-skip-reset-v2-5-c40a8d2d8daa@igalia.com> When a CL/CSD job times out, we check if the GPU has made any progress since the last timeout. If so, instead of resetting the hardware, we skip the reset and allow the timer to be rearmed. This gives long-running jobs a chance to complete. Use the DRM_GPU_SCHED_STAT_NO_HANG status to skip the reset and re-arm the timer. Signed-off-by: Ma?ra Canal --- drivers/gpu/drm/v3d/v3d_sched.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/v3d/v3d_sched.c b/drivers/gpu/drm/v3d/v3d_sched.c index e1997387831541fb053e472672004cf511c25558..fbb09a8aff3740b5cd59573b5f2e26b2ee352dfb 100644 --- a/drivers/gpu/drm/v3d/v3d_sched.c +++ b/drivers/gpu/drm/v3d/v3d_sched.c @@ -761,7 +761,7 @@ v3d_cl_job_timedout(struct drm_sched_job *sched_job, enum v3d_queue q, if (*timedout_ctca != ctca || *timedout_ctra != ctra) { *timedout_ctca = ctca; *timedout_ctra = ctra; - return DRM_GPU_SCHED_STAT_RESET; + return DRM_GPU_SCHED_STAT_NO_HANG; } return v3d_gpu_reset_for_timeout(v3d, sched_job); @@ -805,7 +805,7 @@ v3d_csd_job_timedout(struct drm_sched_job *sched_job) */ if (job->timedout_batches != batches) { job->timedout_batches = batches; - return DRM_GPU_SCHED_STAT_RESET; + return DRM_GPU_SCHED_STAT_NO_HANG; } return v3d_gpu_reset_for_timeout(v3d, sched_job); -- 2.49.0 From mcanal at igalia.com Fri May 30 14:01:35 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Fri, 30 May 2025 11:01:35 -0300 Subject: [PATCH v2 6/8] drm/etnaviv: Use DRM_GPU_SCHED_STAT_NO_HANG to skip the reset In-Reply-To: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> References: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> Message-ID: <20250530-sched-skip-reset-v2-6-c40a8d2d8daa@igalia.com> Etnaviv can skip a hardware reset in two situations: 1. TDR has fired before the free-job worker and the timeout is spurious. 2. The GPU is still making progress on the front-end and we can give the job a chance to complete. Instead of relying on the scheduler internals, use the DRM_GPU_SCHED_STAT_NO_HANG status to skip the reset and re-arm the timer. Signed-off-by: Ma?ra Canal --- drivers/gpu/drm/etnaviv/etnaviv_sched.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/drivers/gpu/drm/etnaviv/etnaviv_sched.c b/drivers/gpu/drm/etnaviv/etnaviv_sched.c index 7146069a98492f5fab2a49d96e2054f649e1fe3d..46f5391e84a12232b247886cf1311f8e09f42f04 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_sched.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_sched.c @@ -40,11 +40,11 @@ static enum drm_gpu_sched_stat etnaviv_sched_timedout_job(struct drm_sched_job int change; /* - * If the GPU managed to complete this jobs fence, the timout is - * spurious. Bail out. + * If the GPU managed to complete this jobs fence, the timeout has + * fired before free-job worker. The timeout is spurious, so bail out. */ if (dma_fence_is_signaled(submit->out_fence)) - goto out_no_timeout; + return DRM_GPU_SCHED_STAT_NO_HANG; /* * If the GPU is still making forward progress on the front-end (which @@ -70,7 +70,7 @@ static enum drm_gpu_sched_stat etnaviv_sched_timedout_job(struct drm_sched_job gpu->hangcheck_dma_addr = dma_addr; gpu->hangcheck_primid = primid; gpu->hangcheck_fence = gpu->completed_fence; - goto out_no_timeout; + return DRM_GPU_SCHED_STAT_NO_HANG; } /* block scheduler */ @@ -86,10 +86,7 @@ static enum drm_gpu_sched_stat etnaviv_sched_timedout_job(struct drm_sched_job drm_sched_resubmit_jobs(&gpu->sched); drm_sched_start(&gpu->sched, 0); - return DRM_GPU_SCHED_STAT_RESET; -out_no_timeout: - list_add(&sched_job->list, &sched_job->sched->pending_list); return DRM_GPU_SCHED_STAT_RESET; } -- 2.49.0 From mcanal at igalia.com Fri May 30 14:01:36 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Fri, 30 May 2025 11:01:36 -0300 Subject: [PATCH v2 7/8] drm/xe: Use DRM_GPU_SCHED_STAT_NO_HANG to skip the reset In-Reply-To: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> References: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> Message-ID: <20250530-sched-skip-reset-v2-7-c40a8d2d8daa@igalia.com> Xe can skip the reset if TDR has fired before the free job worker and can also re-arm the timeout timer in some scenarios. Instead of using the scheduler internals to add the job to the pending list, use the DRM_GPU_SCHED_STAT_NO_HANG status to skip the reset and re-arm the timer. Note that, in the first case, there is no need to restart submission if it hasn't been stopped. Signed-off-by: Ma?ra Canal --- drivers/gpu/drm/xe/xe_guc_submit.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c index 98363d688cbbf884e17e6610366202a3372f5fe0..0149c85aa1a85b2b2e739774310d7b3265e33228 100644 --- a/drivers/gpu/drm/xe/xe_guc_submit.c +++ b/drivers/gpu/drm/xe/xe_guc_submit.c @@ -1067,12 +1067,8 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job) * list so job can be freed and kick scheduler ensuring free job is not * lost. */ - if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &job->fence->flags)) { - xe_sched_add_pending_job(sched, job); - xe_sched_submission_start(sched); - - return DRM_GPU_SCHED_STAT_RESET; - } + if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &job->fence->flags)) + return DRM_GPU_SCHED_STAT_NO_HANG; /* Kill the run_job entry point */ xe_sched_submission_stop(sched); @@ -1247,10 +1243,8 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job) * but there is not currently an easy way to do in DRM scheduler. With * some thought, do this in a follow up. */ - xe_sched_add_pending_job(sched, job); xe_sched_submission_start(sched); - - return DRM_GPU_SCHED_STAT_RESET; + return DRM_GPU_SCHED_STAT_NO_HANG; } static void __guc_exec_queue_fini_async(struct work_struct *w) -- 2.49.0 From mcanal at igalia.com Fri May 30 14:01:37 2025 From: mcanal at igalia.com (=?utf-8?q?Ma=C3=ADra_Canal?=) Date: Fri, 30 May 2025 11:01:37 -0300 Subject: [PATCH v2 8/8] drm/panfrost: Use DRM_GPU_SCHED_STAT_NO_HANG to skip the reset In-Reply-To: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> References: <20250530-sched-skip-reset-v2-0-c40a8d2d8daa@igalia.com> Message-ID: <20250530-sched-skip-reset-v2-8-c40a8d2d8daa@igalia.com> Panfrost can skip the reset if TDR has fired before the free-job worker. Currently, since Panfrost doesn't take any action on these scenarios, the job is being leaked, considering that `free_job()` won't be called. To avoid such leaks, use the DRM_GPU_SCHED_STAT_NO_STAT status to skip the reset and re-arm the timer. Signed-off-by: Ma?ra Canal Reviewed-by: Steven Price --- drivers/gpu/drm/panfrost/panfrost_job.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c index afcffe7f8fe9e11f84e4ab7e8f5a72f7bf583690..842e012cdc68e130a13e08ffae3b7fdf5f8e1acc 100644 --- a/drivers/gpu/drm/panfrost/panfrost_job.c +++ b/drivers/gpu/drm/panfrost/panfrost_job.c @@ -751,11 +751,11 @@ static enum drm_gpu_sched_stat panfrost_job_timedout(struct drm_sched_job int js = panfrost_job_get_slot(job); /* - * If the GPU managed to complete this jobs fence, the timeout is - * spurious. Bail out. + * If the GPU managed to complete this jobs fence, the timeout has + * fired before free-job worker. The timeout is spurious, so bail out. */ if (dma_fence_is_signaled(job->done_fence)) - return DRM_GPU_SCHED_STAT_RESET; + return DRM_GPU_SCHED_STAT_NO_HANG; /* * Panfrost IRQ handler may take a long time to process an interrupt @@ -770,7 +770,7 @@ static enum drm_gpu_sched_stat panfrost_job_timedout(struct drm_sched_job if (dma_fence_is_signaled(job->done_fence)) { dev_warn(pfdev->dev, "unexpectedly high interrupt latency\n"); - return DRM_GPU_SCHED_STAT_RESET; + return DRM_GPU_SCHED_STAT_NO_HANG; } dev_err(pfdev->dev, "gpu sched timeout, js=%d, config=0x%x, status=0x%x, head=0x%x, tail=0x%x, sched_job=%p", -- 2.49.0