<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="ZH-CN" link="#0563C1" vlink="#954F72" style="text-justify-trim:punctuation"> <div class="WordSection1"> + david<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Liu, Monk Sent: Wednesday, October 11, 2017 1:34 PM To: Koenig, Christian <Christian.Koenig@amd.com>; Haehnle, Nicolai <Nicolai.Haehnle@amd.com>; Olsak, Marek <Marek.Olsak@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com> Cc: amd-gfx@lists.freedesktop.org; Ding, Pixel <Pixel.Ding@amd.com>; Jiang, Jerry (SW) <Jerry.Jiang@amd.com>; Li, Bingley (Bingley.Li@amd.com) <Bingley.Li@amd.com>; Ramirez, Alejandro <Alejandro.Ramirez@amd.com>; Filipas, Mario <Mario.Filipas@amd.com> Subject: TDR and VRAM lost handling in KMD:<o:p></o:p> </div> </div> <o:p> </o:p> Hi Christian & Nicolai,<o:p></o:p> <o:p> </o:p> We need to achieve some agreements on what should MESA/UMD do and what should KMD do, please give your comments with “okay” or “No” and your idea on below items,<o:p></o:p> <o:p> </o:p> <![if !supportLists]>l <![endif]>When a job timed out (set from lockup_timeout kernel parameter), What KMD should do in TDR routine :<o:p></o:p> <o:p> </o:p> <![if !supportLists]>1. <![endif]>Update adev->gpu_reset_counter, and stop scheduler first, (gpu_reset_counter is used to force vm flush after GPU reset, out of this thread’s scope so no more discussion on it)<o:p></o:p> <![if !supportLists]>2. <![endif]>Set its fence error status to “ETIME”,<o:p></o:p> <![if !supportLists]>3. <![endif]>Find the entity/ctx behind this job, and set this ctx as “guilty”<o:p></o:p> <![if !supportLists]>4. <![endif]>Kick out this job from scheduler’s mirror list, so this job won’t get re-scheduled to ring anymore.<o:p></o:p> <![if !supportLists]>5. <![endif]>Kick out all jobs in this “guilty” ctx’s KFIFO queue, and set all their fence status to “ECANCELED”<o:p></o:p> <![if !supportLists]>6. <![endif]>Force signal all fences that get kicked out by above two steps, otherwise UMD will block forever if waiting on those fences<o:p></o:p> <![if !supportLists]>7. <![endif]>Do gpu reset, which is can be some callbacks to let bare-metal and SR-IOV implement with their favor style <o:p></o:p> <![if !supportLists]>8. <![endif]>After reset, KMD need to aware if the VRAM lost happens or not, bare-metal can implement some function to judge, while for SR-IOV I prefer to read it from GIM side (for initial version we consider it’s always VRAM lost, till GIM side change aligned)<o:p></o:p> <![if !supportLists]>9. <![endif]>If VRAM lost not hit, continue, otherwise:<o:p></o:p> <![if !supportLists]>a) <![endif]>Update adev->vram_lost_counter,<o:p></o:p> <![if !supportLists]>b) <![endif]>Iterate over all living ctx, and set all ctx as “guilty” since VRAM lost actually ruins all VRAM contents<o:p></o:p> <![if !supportLists]>c) <![endif]>Kick out all jobs in all ctx’s KFIFO queue, and set all their fence status to “ECANCELDED”<o:p></o:p> <![if !supportLists]>10. <![endif]>Do GTT recovery and VRAM page tables/entries recovery (optional, do we need it ???)<o:p></o:p> <![if !supportLists]>11. <![endif]>Re-schedule all JOBs remains in mirror list to ring again and restart scheduler (for VRAM lost case, no JOB will re-scheduled)<o:p></o:p> <o:p> </o:p> <![if !supportLists]>l <![endif]>For cs_wait() IOCTL:<o:p></o:p> After it found fence signaled, it should check with “dma_fence_get_status” to see if there is error there,<o:p></o:p> And return the error status of fence<o:p></o:p> <o:p> </o:p> <![if !supportLists]>l <![endif]>For cs_wait_fences() IOCTL:<o:p></o:p> Similar with above approach<o:p></o:p> <o:p> </o:p> <![if !supportLists]>l <![endif]>For cs_submit() IOCTL:<o:p></o:p> It need to check if current ctx been marked as “guilty” and return “ECANCELED” if so<o:p></o:p> <o:p> </o:p> <![if !supportLists]>l <![endif]>Introduce a new IOCTL to let UMD query vram_lost_counter:<o:p></o:p> This way, UMD can also block app from submitting, like @Nicolai mentioned, we can cache one copy of vram_lost_counter when enumerate physical device, and deny all <o:p></o:p> gl-context from submitting if the counter queried bigger than that one cached in physical device. (looks a little overkill to me, but easy to implement ) <o:p></o:p> UMD can also return error to APP when creating gl-context if found current queried vram_lost_counter bigger than that one cached in physical device.<o:p></o:p> <o:p> </o:p> BTW: I realized that gl-context is a little different with kernel’s context. Because for kernel. BO is not related with context but only with FD, while in UMD, BO have a backend<o:p></o:p> gl-context, so block submitting in UMD layer is also needed although KMD will do its job as bottom line <o:p></o:p> <o:p> </o:p> <![if !supportLists]>l <![endif]>Basically “vram_lost_counter” is exposure by kernel to let UMD take the control of robust extension feature, it will be UMD’s call to move, KMD only deny “guilty” context from submitting<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Need your feedback, thx<o:p></o:p> <o:p> </o:p> We’d better make TDR feature landed ASAP<o:p></o:p> <o:p> </o:p> BR Monk<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> </div> </body> </html>