[PATCH v3.1] drm/xe: Implement VM snapshot support for BO's and userptr

Tue Feb 13 14:41:59 UTC 2024

On 2024-02-06 21:08, Souza, Jose wrote:
> On Mon, 2024-02-05 at 12:35 -0800, José Roberto de Souza wrote:
>> On Mon, 2024-02-05 at 08:32 -0800, José Roberto de Souza wrote:
>>> On Fri, 2024-02-02 at 23:45 +0100, Maarten Lankhorst wrote:
>>>> Since we cannot immediately capture the BO's and userptr, perform it in
>>>> 2 stages. The immediate stage takes a reference to each BO and userptr,
>>>> while a delayed worker captures the contents and then frees the
>>>> reference.
>>>>
>>>> This is required because in signaling context, no locks can be taken, no
>>>> memory can be allocated, and no waits on userspace can be performed.
>>>>
>>>> With the delayed worker, all of this can be performed very easily,
>>>> without having to resort to hacks.
>>>>
>>>> Changes since v1:
>>>> - Fix crash on NULL captured vm.
>>>> - Use ascii85_encode to capture BO contents and save some space.
>>>> - Add length to coredump output for each captured area.
>>>> Changes since v2:
>>>> - Dump each mapping on their own line, to simplify tooling.
>>>> - Fix null pointer deref in xe_vm_snapshot_free.
>>>
>>>
>>> First crash dump with piglit already got:
>>>
>>>
>>> [   65.097735] xe 0000:00:02.0: [drm:intel_pps_vdd_off_sync_unlocked [xe]] [ENCODER:307:DDI A/PHY A] PPS 0 PP_STATUS: 0x80000008 PP_CONTROL:
>>> 0x00000067
>>> [   65.238975] xe 0000:00:02.0: [drm:intel_power_well_disable [xe]] disabling PW_3
>>> [   65.239050] xe 0000:00:02.0: [drm:intel_power_well_disable [xe]] disabling PW_2
>>> [  305.225363] loop0: detected capacity change from 0 to 8
>>> [  430.689574] xe 0000:00:02.0: [drm] Timedout job: seqno=4294967169, guc_id=2, flags=0x8
>>> [  430.689933] ------------[ cut here ]------------
>>> [  430.689958] DEBUG_LOCKS_WARN_ON(lock->magic != lock)
>>> [  430.689964] WARNING: CPU: 6 PID: 94 at kernel/locking/mutex.c:587 __mutex_lock+0x50d/0xb80
>>> [  430.690007] Modules linked in: snd_hda_codec_hdmi snd_ctl_led ledtrig_audio snd_hda_codec_realtek snd_hda_codec_generic xe drm_ttm_helper gpu_sched
>>> drm_suballoc_helper drm_gpuvm drm_exec i2c_algo_bit drm_buddy drm_display_helper ttm x86_pkg_temp_thermal mei_pxp mei_hdcp coretemp snd_hda_intel
>>> wmi_bmof crct10dif_pclmul snd_intel_dspcfg crc32_pclmul snd_hda_codec e1000e video ghash_clmulni_intel snd_hwdep kvm_intel snd_hda_core ptp i2c_i801
>>> snd_pcm pps_core i2c_smbus mei_me mei intel_pmc_core intel_vsec wmi pmt_telemetry pmt_class fuse
>>> [  430.690164] CPU: 6 PID: 94 Comm: kworker/u16:2 Not tainted 6.8.0-rc3-zeh-xe+ #1222
>>> [  430.690188] Hardware name: Dell Inc. Latitude 5420/01M3M4, BIOS 1.27.0 03/17/2023
>>> [  430.690212] Workqueue: gt-ordered-wq drm_sched_job_timedout [gpu_sched]
>>> [  430.690239] RIP: 0010:__mutex_lock+0x50d/0xb80
>>> [  430.690254] Code: ff 85 c0 0f 84 7d fb ff ff 8b 15 b2 ec ba 00 85 d2 0f 85 6f fb ff ff 48 c7 c6 ca 46 3a 82 48 c7 c7 73 d6 39 82 e8 73 19 40 ff
>>> <0f> 0b e9 55 fb ff ff 31 c9 31 d2 4c 89 e7 e8 80 7e 47 ff 84 c0 0f
>>> [  430.690298] RSP: 0018:ffffc900005ebc60 EFLAGS: 00010282
>>> [  430.690313] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
>>> [  430.690332] RDX: 0000000000000002 RSI: 0000000000000027 RDI: 00000000ffffffff
>>> [  430.690356] RBP: ffffc900005ebcf0 R08: 00000000fffeffff R09: 0000000000000001
>>> [  430.690375] R10: 00000000fffeffff R11: ffff888287080000 R12: ffff88824998d4a8
>>> [  430.690394] R13: 0000000000000000 R14: ffff88824998d038 R15: ffff8881109b3e00
>>> [  430.690417] FS:  0000000000000000(0000) GS:ffff888287b00000(0000) knlGS:0000000000000000
>>> [  430.690438] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [  430.690455] CR2: 00007f5ccd99de3e CR3: 000000000564a003 CR4: 0000000000770ef0
>>> [  430.690475] PKRU: 55555554
>>> [  430.690485] Call Trace:
>>> [  430.690495]  <TASK>
>>> [  430.690504]  ? __mutex_lock+0x50d/0xb80
>>> [  430.690518]  ? __warn+0x7c/0x170
>>> [  430.690538]  ? __mutex_lock+0x50d/0xb80
>>> [  430.690575]  ? report_bug+0x189/0x1c0
>>> [  430.690597]  ? handle_bug+0x36/0x70
>>> [  430.690616]  ? exc_invalid_op+0x13/0x60
>>> [  430.690637]  ? asm_exc_invalid_op+0x16/0x20
>>> [  430.690661]  ? __mutex_lock+0x50d/0xb80
>>> [  430.690680]  ? __slab_alloc.isra.0+0x4d/0x90
>>> [  430.690703]  ? __slab_alloc.isra.0+0x5a/0x90
>>> [  430.690722]  ? xe_vm_snapshot_capture+0x35/0x1f0 [xe]
>>> [  430.690840]  ? rcu_is_watching+0xd/0x40
>>> [  430.690863]  ? __kmalloc+0x2bd/0x400
>>> [  430.690883]  ? xe_vm_snapshot_capture+0x35/0x1f0 [xe]
>>> [  430.690984]  xe_vm_snapshot_capture+0x35/0x1f0 [xe]
>>> [  430.691070]  ? xe_sched_job_snapshot_capture+0x64/0x80 [xe]
>>> [  430.691155]  xe_devcoredump+0x1b9/0x2e0 [xe]
>>> [  430.691226]  guc_exec_queue_timedout_job+0x98/0x5a0 [xe]
>>> [  430.691305]  drm_sched_job_timedout+0x77/0xe0 [gpu_sched]
>>> [  430.691337]  ? process_one_work+0x18d/0x4d0
>>> [  430.691360]  process_one_work+0x1f4/0x4d0
>>> [  430.691378]  worker_thread+0x1d8/0x3c0
>>> [  430.691396]  ? rescuer_thread+0x390/0x390
>>> [  430.691414]  kthread+0xfb/0x130
>>> [  430.691429]  ? kthread_complete_and_exit+0x20/0x20
>>> [  430.691451]  ret_from_fork+0x28/0x40
>>> [  430.691470]  ? kthread_complete_and_exit+0x20/0x20
>>> [  430.691495]  ret_from_fork_asm+0x11/0x20
>>> [  430.691514]  </TASK>
>>> [  430.691523] irq event stamp: 6077281
>>> [  430.691537] hardirqs last  enabled at (6077281): [<ffffffff81d2dcaa>] _raw_spin_unlock_irqrestore+0x4a/0x70
>>> [  430.691598] hardirqs last disabled at (6077280): [<ffffffff81d2da7a>] _raw_spin_lock_irqsave+0x4a/0x50
>>> [  430.691640] softirqs last  enabled at (6075212): [<ffffffff81131962>] irq_exit_rcu+0x82/0xe0
>>> [  430.691679] softirqs last disabled at (6075205): [<ffffffff81131962>] irq_exit_rcu+0x82/0xe0
>>> [  430.691722] ---[ end trace 0000000000000000 ]---
>>> [  430.691802] xe 0000:00:02.0: [drm] Xe device coredump has been created
>>
>> Hold a bit in this crash, the VM is being destroyed by Iris before drm_exec is done.
>> I think fixing it will fix this warning.
> 
> Yep that was the case, it will be fixed by https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27500

It will fix the warning, but not the kernel bug. snap_mutex should be 
left valid for longer instead.

I'll destroy it from vm_destroy_work_func instead of xe_vm_close_and_put.

Cheers,
~Maarten