[PATCH] drm/amdgpu: Add timeout for sync wait

[Felix Kuehling]

On 2023-10-19 05:31, Emily Deng wrote:
> Issue: Dead heappen during gpu recover
>
> [56433.829492] amdgpu 0000:04:00.0: amdgpu: GPU reset begin!
> [56550.499625] INFO: task kworker/u80:0:10 blocked for more than 120 seconds.
> [56550.520215]       Tainted: G           OE      6.2.0-34-generic #34~22.04.1-Ubuntu
> [56550.542883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [56550.566313] task:kworker/u80:0   state:D stack:0     pid:10    ppid:2      flags:0x00004000
> [56550.591318] Workqueue: kfd_restore_wq restore_process_worker [amdgpu]
> [56550.611391] Call Trace:
> [56550.618698]  <TASK>
> [56550.624968]  __schedule+0x2b7/0x5f0
> [56550.635416]  schedule+0x68/0x110
> [56550.645090]  schedule_timeout+0x151/0x160
> [56550.657096]  ? amdgpu_vm_bo_update+0x46e/0x660 [amdgpu]
> [56550.673245]  dma_fence_default_wait+0x1a2/0x1e0
> [56550.686818]  ? __pfx_dma_fence_default_wait_cb+0x10/0x10
> [56550.702728]  dma_fence_wait_timeout+0x117/0x140
> [56550.716301]  amdgpu_sync_wait+0x62/0xa0 [amdgpu]
> [56550.730654]  amdgpu_amdkfd_gpuvm_restore_process_bos+0x59e/0x770 [amdgpu]
> [56550.751668]  ? newidle_balance+0x298/0x490
> [56550.763936]  restore_process_worker+0x42/0x270 [amdgpu]
> [56550.780183]  process_one_work+0x21f/0x440
> [56550.792193]  worker_thread+0x50/0x3f0
> [56550.803165]  ? __pfx_worker_thread+0x10/0x10
> [56550.815934]  kthread+0xee/0x120
> [56550.825342]  ? __pfx_kthread+0x10/0x10
> [56550.836548]  ret_from_fork+0x2c/0x50
> [56550.847262]  </TASK>
> [ 1935.215502] Call Trace:
> [ 1935.222827]  <TASK>
> [ 1935.229121]  __schedule+0x23d/0x5a0
> [ 1935.239583]  schedule+0x4e/0xc0
> [ 1935.248983]  schedule_timeout+0x103/0x140
> [ 1935.261002]  __wait_for_common+0xae/0x150
> [ 1935.273008]  ? usleep_range_state+0x90/0x90
> [ 1935.285546]  wait_for_completion+0x24/0x30
> [ 1935.297813]  __flush_work.isra.0+0x175/0x280
> [ 1935.310611]  ? worker_detach_from_pool+0xc0/0xc0
> [ 1935.324436]  flush_delayed_work+0x31/0x50
> [ 1935.336455]  kfd_suspend_all_processes+0x96/0x150 [amdgpu]
> [ 1935.353429]  kgd2kfd_suspend+0xb8/0xe0 [amdgpu]
> [ 1935.367469]  kgd2kfd_pre_reset+0x81/0xf0 [amdgpu]
> [ 1935.382036]  amdgpu_amdkfd_pre_reset+0x1a/0x30 [amdgpu]
> [ 1935.398156]  amdgpu_device_gpu_recover.cold+0x210/0xcf2 [amdgpu]
> [ 1935.416722]  amdgpu_job_timedout+0x19f/0x1e0 [amdgpu]
> [ 1935.432367]  drm_sched_job_timedout+0x6f/0x120 [amd_sched]
> [ 1935.448792]  process_one_work+0x22b/0x3d0
> [ 1935.460806]  worker_thread+0x53/0x420
> [ 1935.471777]  ? process_one_work+0x3d0/0x3d0
> [ 1935.484307]  kthread+0x12a/0x150
> [ 1935.493993]  ? set_kthread_struct+0x50/0x50
> [ 1935.506513]  ret_from_fork+0x22/0x30

Looking at the time stamps, this seems to be a mash-up of two different 
logs. I think you're trying to show how a restore_processes worker is 
stuck on a fence, and that's causing kgd2kfd_pre_reset to hang when it 
tries to flush the work.

The fence it's hanging on is probably something related to a page table 
update that got caught up in the GPU hang. Adding a timeout here seems 
reasonable. There may be another problem, because 
amdgpu_amdkfd_gpuvm_restore_process_bos ignores the return value of 
amdgpu_sync_wait. We shouldi probably handle the timeout gracefully with 
a "goto validate_map_fail".

Regards,
   Felix


>
> It is because the amdgpu_sync_wait is waiting for the bad job's fence, and
> never return, so the recover couldn't continue.
>
>
> Signed-off-by: Emily Deng <Emily.Deng at amd.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu_sync.c | 16 +++++++++++++---
>   1 file changed, 13 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_sync.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_sync.c
> index dcd8c066bc1f..c922867c5675 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_sync.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_sync.c
> @@ -406,9 +406,19 @@ int amdgpu_sync_wait(struct amdgpu_sync *sync, bool intr)
>   	int i, r;
>   
>   	hash_for_each_safe(sync->fences, i, tmp, e, node) {
> -		r = dma_fence_wait(e->fence, intr);
> -		if (r)
> -			return r;
> +		struct drm_sched_fence *s_fence = to_drm_sched_fence(e->fence);
> +
> +		if (s_fence) {
> +			r = dma_fence_wait_timeout(e->fence, intr, s_fence->sched->timeout);
> +			if (r == 0)
> +				r = -ETIMEDOUT;
> +			if (r < 0)
> +				return r;
> +		} else {
> +			r = dma_fence_wait(e->fence, intr);
> +			if (r)
> +				return r;
> +		}
>   
>   		amdgpu_sync_entry_free(e);
>   	}