<html><head> <meta http-equiv="Content-Type" content="text/html; charset=Windows-1252"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">There is a misunderstanding here: <blockquote type="cite"> Did you find out why the zero refcount on the finished fence happens before the fence was signaled ?</blockquote> The refcount on the finished fence doesn't become zero before it is signaled, it becomes zero while it is signaled. CPU 1 calls dma_fence_signal(fence) without holding a reference to the fence. CPU 2 at the same time checks if the fence is signaled and frees the last reference because it find the signaled flag to be set. The problem is now that dma_fence_signal() wants to set the timestamp after setting the signaled flag and now races with freeing the memory. That's a really hard to hit problem, but it indeed seems to be possible. Christian. Am 24.03.20 um 15:52 schrieb Andrey Grodzovsky: </div> <blockquote type="cite" cite="mid:be0e40cf-3ecf-ebe8-2d73-1dd937450c18@amd.com"> This is only for the guilty job which was removed from the ring_mirror_list due to completion and hence will not be resubmitted by recovery and will not be freed by the usual flow in drm_sched_get_cleanup_job (see drm_sched_stop) Andrey <div class="moz-cite-prefix">On 3/24/20 10:45 AM, Pan, Xinhui wrote: </div> <blockquote type="cite" cite="mid:SN6PR12MB2800A5049C6AB62B7A002AC987F10@SN6PR12MB2800.namprd12.prod.outlook.com"> [AMD Official Use Only - Internal Distribution Only] <div> <div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; "> Does this issue occur when gpu recovery? </div> <div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; "> I just check the code, fence timedout will free job and put its fence. but gpu recovery might resubmit job. </div> <div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; "> Correct me if I am wrong.</div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr">From: amd-gfx <a class="moz-txt-link-rfc2396E" href="mailto:amd-gfx-bounces@lists.freedesktop.org" moz-do-not-send="true"><amd-gfx-bounces@lists.freedesktop.org></a> on behalf of Andrey Grodzovsky <a class="moz-txt-link-rfc2396E" href="mailto:Andrey.Grodzovsky@amd.com" moz-do-not-send="true"><Andrey.Grodzovsky@amd.com></a> Sent: Tuesday, March 24, 2020 11:40:06 AM To: Tao, Yintian <a class="moz-txt-link-rfc2396E" href="mailto:Yintian.Tao@amd.com" moz-do-not-send="true"><Yintian.Tao@amd.com></a>; Koenig, Christian <a class="moz-txt-link-rfc2396E" href="mailto:Christian.Koenig@amd.com" moz-do-not-send="true"><Christian.Koenig@amd.com></a>; Deucher, Alexander <a class="moz-txt-link-rfc2396E" href="mailto:Alexander.Deucher@amd.com" moz-do-not-send="true"><Alexander.Deucher@amd.com></a> Cc: <a class="moz-txt-link-abbreviated" href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">amd-gfx@lists.freedesktop.org</a> <a class="moz-txt-link-rfc2396E" href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true"><amd-gfx@lists.freedesktop.org></a> Subject: Re: [PATCH] drm/amdgpu: hold the reference of finished fence <div> </div> </div> <div class="BodyFragment"> <div class="PlainText"> On 3/23/20 10:22 AM, Yintian Tao wrote: > There is one one corner case at dma_fence_signal_locked > which will raise the NULL pointer problem just like below. > ->dma_fence_signal > ->dma_fence_signal_locked > ->test_and_set_bit > here trigger dma_fence_release happen due to the zero of fence refcount. Did you find out why the zero refcount on the finished fence happens before the fence was signaled ? The finished fence is created with refcount set to 1 in drm_sched_fence_create->dma_fence_init and then the refcount is decremented in drm_sched_main->amdgpu_job_free_cb->drm_sched_job_cleanup. This should only happen after fence is already signaled (see drm_sched_get_cleanup_job). On top of that the finished fence is referenced from other places (e.g. entity->last_scheduled e.t.c)... > > ->dma_fence_put > ->dma_fence_release > ->drm_sched_fence_release_scheduled > ->call_rcu > here make the union fled “cb_list” at finished fence > to NULL because struct rcu_head contains two pointer > which is same as struct list_head cb_list > > Therefore, to hold the reference of finished fence at drm_sched_process_job > to prevent the null pointer during finished fence dma_fence_signal > > [ 732.912867] BUG: kernel NULL pointer dereference, address: 0000000000000008 > [ 732.914815] #PF: supervisor write access in kernel mode > [ 732.915731] #PF: error_code(0x0002) - not-present page > [ 732.916621] PGD 0 P4D 0 > [ 732.917072] Oops: 0002 [#1] SMP PTI > [ 732.917682] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G OE 5.4.0-rc7 #1 > [ 732.918980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 > [ 732.920906] RIP: 0010:dma_fence_signal_locked+0x3e/0x100 > [ 732.938569] Call Trace: > [ 732.939003] <IRQ> > [ 732.939364] dma_fence_signal+0x29/0x50 > [ 732.940036] drm_sched_fence_finished+0x12/0x20 [gpu_sched] > [ 732.940996] drm_sched_process_job+0x34/0xa0 [gpu_sched] > [ 732.941910] dma_fence_signal_locked+0x85/0x100 > [ 732.942692] dma_fence_signal+0x29/0x50 > [ 732.943457] amdgpu_fence_process+0x99/0x120 [amdgpu] > [ 732.944393] sdma_v4_0_process_trap_irq+0x81/0xa0 [amdgpu] > > v2: hold the finished fence at drm_sched_process_job instead of > amdgpu_fence_process > v3: resume the blank line > > Signed-off-by: Yintian Tao <a class="moz-txt-link-rfc2396E" href="mailto:yttao@amd.com" moz-do-not-send="true"><yttao@amd.com></a> > --- > drivers/gpu/drm/scheduler/sched_main.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c > index a18eabf692e4..8e731ed0d9d9 100644 > --- a/drivers/gpu/drm/scheduler/sched_main.c > +++ b/drivers/gpu/drm/scheduler/sched_main.c > @@ -651,7 +651,9 @@ static void drm_sched_process_job(struct dma_fence *f, struct dma_fence_cb *cb) > > trace_drm_sched_process_job(s_fence); > > + dma_fence_get(&s_fence->finished); > drm_sched_fence_finished(s_fence); If the fence was already released during call to drm_sched_fence_finished->dma_fence_signal->... why is it safe to reference the s_fence just before that call ? Can't it already be released by this time ? Andrey > + dma_fence_put(&s_fence->finished); > wake_up_interruptible(&sched->wake_up_worker); > } > _______________________________________________ amd-gfx mailing list <a class="moz-txt-link-abbreviated" href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">amd-gfx@lists.freedesktop.org</a> <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Cxinhui.pan%40amd.com%7C65933fca0b414d12aab408d7cfa51165%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637206180230440562&sdata=z6ec%2BcWkwjaDgZvkpL3jOMYkBtDjbNOxlXiAk4Ri5Ck%3D&reserved=0" moz-do-not-send="true">https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Cxinhui.pan%40amd.com%7C65933fca0b414d12aab408d7cfa51165%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637206180230440562&sdata=z6ec%2BcWkwjaDgZvkpL3jOMYkBtDjbNOxlXiAk4Ri5Ck%3D&reserved=0</a> </div> </div> </div> </blockquote> </blockquote> </body> </html>