BUG: KASAN: slab-use-after-free in drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]

Tue Mar 21 18:47:44 UTC 2023

Hi Mikhail,

Am 20.03.23 um 13:05 schrieb Mikhail Gavrilov:
> Hi,
> after enabling KASAN literally I was bombarded with messages about
> slab-use-after-free in drm_sched_get_cleanup_job.

mhm, interesting.

> All messages has similar backtrace:
> [ 1138.492091] ==================================================================
> [ 1138.492104] BUG: KASAN: slab-use-after-free in
> drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492120] Read of size 8 at addr ffff88815adf04c0 by task sdma1/749
>
> [ 1138.492130] CPU: 29 PID: 749 Comm: sdma1 Tainted: G        W    L
>   -------  ---  6.3.0-0.rc2.20230317git38e04b3e4240.27.fc39.x86_64+debug
> #1
> [ 1138.492136] Hardware name: System manufacturer System Product
> Name/ROG STRIX X570-I GAMING, BIOS 4601 02/02/2023
> [ 1138.492141] Call Trace:
> [ 1138.492145]  <TASK>
> [ 1138.492150]  dump_stack_lvl+0x72/0xc0
> [ 1138.492159]  print_report+0xcf/0x670
> [ 1138.492169]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492181]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492193]  kasan_report+0xa4/0xe0
> [ 1138.492200]  ? drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492215]  drm_sched_get_cleanup_job+0x47b/0x5c0 [gpu_sched]
> [ 1138.492229]  drm_sched_main+0x643/0x990 [gpu_sched]
> [ 1138.492245]  ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
> [ 1138.492259]  ? __pfx_autoremove_wake_function+0x10/0x10
> [ 1138.492269]  ? __kthread_parkme+0xc1/0x1f0
> [ 1138.492277]  ? __pfx_drm_sched_main+0x10/0x10 [gpu_sched]
> [ 1138.492288]  kthread+0x29e/0x340
> [ 1138.492294]  ? __pfx_kthread+0x10/0x10
> [ 1138.492301]  ret_from_fork+0x2c/0x50
> [ 1138.492314]  </TASK>
>
> [ 1138.492320] Allocated by task 10867:
> [ 1138.492323]  kasan_save_stack+0x2f/0x50
> [ 1138.492329]  kasan_set_track+0x21/0x30
> [ 1138.492334]  __kasan_kmalloc+0x8b/0x90
> [ 1138.492339]  amdgpu_driver_open_kms+0x10b/0x5a0 [amdgpu]
> [ 1138.493112]  drm_file_alloc+0x46e/0x880
> [ 1138.493120]  drm_open_helper+0x161/0x460
> [ 1138.493126]  drm_open+0x1e7/0x5c0
> [ 1138.493131]  drm_stub_open+0x24d/0x400
> [ 1138.493138]  chrdev_open+0x215/0x620
> [ 1138.493144]  do_dentry_open+0x5f1/0x1000
> [ 1138.493149]  path_openat+0x1b3d/0x28a0
> [ 1138.493156]  do_filp_open+0x1bd/0x400
> [ 1138.493161]  do_sys_openat2+0x140/0x420
> [ 1138.493167]  __x64_sys_openat+0x11f/0x1d0
> [ 1138.493173]  do_syscall_64+0x5b/0x80
> [ 1138.493179]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> [ 1138.493189] Freed by task 10867:
> [ 1138.493193]  kasan_save_stack+0x2f/0x50
> [ 1138.493199]  kasan_set_track+0x21/0x30
> [ 1138.493205]  kasan_save_free_info+0x2a/0x50
> [ 1138.493210]  __kasan_slab_free+0x107/0x1a0
> [ 1138.493216]  slab_free_freelist_hook+0x11e/0x1d0
> [ 1138.493221]  __kmem_cache_free+0xbc/0x2e0
> [ 1138.493227]  amdgpu_driver_postclose_kms+0x582/0x8d0 [amdgpu]
> [ 1138.493963]  drm_file_free.part.0+0x638/0xb70
> [ 1138.493969]  drm_release+0x1ea/0x470
> [ 1138.493975]  __fput+0x213/0x9e0
> [ 1138.493981]  task_work_run+0x11b/0x200
> [ 1138.493987]  exit_to_user_mode_prepare+0x23a/0x260
> [ 1138.493994]  syscall_exit_to_user_mode+0x16/0x50
> [ 1138.494001]  do_syscall_64+0x67/0x80
> [ 1138.494006]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
>
> [ 1138.494015] The buggy address belongs to the object at ffff88815adf0000
>                  which belongs to the cache kmalloc-4k of size 4096
> [ 1138.494021] The buggy address is located 1216 bytes inside of
>                  freed 4096-byte region [ffff88815adf0000, ffff88815adf1000)
>
> [ 1138.494030] The buggy address belongs to the physical page:
> [ 1138.494034] page:0000000017cd5a82 refcount:1 mapcount:0
> mapping:0000000000000000 index:0x0 pfn:0x15adf0
> [ 1138.494041] head:0000000017cd5a82 order:3 entire_mapcount:0
> nr_pages_mapped:0 pincount:0
> [ 1138.494046] flags:
> 0x17ffffc0010200(slab|head|node=0|zone=2|lastcpupid=0x1fffff)
> [ 1138.494055] raw: 0017ffffc0010200 ffff88810004d040 dead000000000122
> 0000000000000000
> [ 1138.494061] raw: 0000000000000000 0000000000040004 00000001ffffffff
> 0000000000000000
> [ 1138.494065] page dumped because: kasan: bad access detected
>
> [ 1138.494071] Memory state around the buggy address:
> [ 1138.494076]  ffff88815adf0380: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494080]  ffff88815adf0400: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494084] >ffff88815adf0480: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494088]                                            ^
> [ 1138.494092]  ffff88815adf0500: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494095]  ffff88815adf0580: fb fb fb fb fb fb fb fb fb fb fb fb
> fb fb fb fb
> [ 1138.494099] ==================================================================
> [ 1138.494140] Disabling lock debugging due to kernel taint
>
> First thing I checked RAM for sure that it is definitely not a
> hardware problem. I started investigating which application triggered
> this message. I started to notice that it usually happens when any
> game starts in the steam client. So I came to the culprit of the
> problem is the compilation of shaders. For reproduction we should have
> a kernel with enabled KASAN and a steam client. After allowing
> background processing of Vulkan shaders the issue happens.
> Unfortunately it does not happen immediately as you may expect.
> I record four screencasts and uploaded them on to youtube [1-4] for
> proving that definitely processing of Vulkan shaders  culprit here but
> this may happens after some time.
>
> [1] https://youtu.be/hFuWlGTbetg
> [2] https://youtu.be/TAOqoYUYA5Q
> [3] https://youtu.be/9SZNXWhwAZs
> [4] https://youtu.be/W9gtYyL-2pg
>
> I do not know which information would be also useful. Please ask me.
>
> I attached a full kernel log and config if someone would be interested
> to see it.

That looks like a reference counting issue to me.

I'm going to take a look, but we have already fixed one of those recently.

Probably best that you try this on drm-fixes, just to double check that 
this isn't the same issue.

Thanks,
Christian.