<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix"> <blockquote type="cite"> [ML] I think context is better than entity, because for example if you only block entity_0 of context and allow entity_N run, that means the dependency between entities are broken (e.g. page table updates in Sdma entity pass but gfx submit in GFX entity blocked, not make sense to me) We’d better either block the whole context or let not… </blockquote> Page table updates are not part of any context. So I think the only thing we can do is to mark the entity as not scheduled any more. <blockquote type="cite"> 1. Kick out all jobs in this “guilty” ctx’s KFIFO queue, and set all their fence status to “ECANCELED” Setting ECANCELED should be ok. But I think we should do this when we try to run the jobs and not during GPU reset. [ML] without deep thought and expritment, I’m not sure the difference between them, but kick it out in gpu_reset routine is more efficient, </blockquote> I really don't think so. Kicking them out during gpu_reset sounds racy to me once more. And marking them canceled when we try to run them has the clear advantage that all dependencies are meet first. <blockquote type="cite"> ML: KMD mark all contexts as guilty is because that way we can unify our IOCTL behavior: e.g. for IOCTL only block “guilty”context , no need to worry about vram-lost-counter anymore, that’s a implementation style. I don’t think it is related with UMD layer, </blockquote> I don't think that this is a good idea. Instead when you want to unify the behavior we should use the vram_lost_counter as marker for the guilty context. Regards, Christian. Am 11.10.2017 um 10:48 schrieb Liu, Monk: </div> <blockquote type="cite" cite="mid:BLUPR12MB0449287A92DF8D3EB30BE6A6844A0@BLUPR12MB0449.namprd12.prod.outlook.com"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)">  <style></style> <div class="WordSection1"> <o:p> </o:p> On "guilty": "guilty" is a term that's used by APIs (e.g. OpenGL), so it's reasonable to use it. However, it does not make sense to mark idle contexts as "guilty" just because VRAM is lost. VRAM lost is a perfect example where the driver should report context lost to applications with the "innocent" flag for contexts that were idle at the time of reset. The only context(s) that should be reported as "guilty" (or perhaps "unknown" in some cases) are the ones that were executing at the time of reset.<o:p></o:p> <o:p> </o:p> ML: KMD mark all contexts as guilty is because that way we can unify our IOCTL behavior: e.g. for IOCTL only block “guilty”context , no need to worry about vram-lost-counter anymore, that’s a implementation style. I don’t think it is related with UMD layer,<o:p></o:p> For UMD the gl-context isn’t aware of by KMD, so UMD can implement it own “guilty” gl-context if you want.<o:p></o:p> <o:p> </o:p> If KMD doesn’t mark all ctx as guilty after VRAM lost, can you illustrate what rule KMD should obey to check in KMS IOCTL like cs_sumbit ?? let’s see which way better <o:p></o:p> <o:p> </o:p> <o:p> </o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Haehnle, Nicolai Sent: Wednesday, October 11, 2017 4:41 PM To: Liu, Monk <a class="moz-txt-link-rfc2396E" href="mailto:Monk.Liu@amd.com"><Monk.Liu@amd.com></a>; Koenig, Christian <a class="moz-txt-link-rfc2396E" href="mailto:Christian.Koenig@amd.com"><Christian.Koenig@amd.com></a>; Olsak, Marek <a class="moz-txt-link-rfc2396E" href="mailto:Marek.Olsak@amd.com"><Marek.Olsak@amd.com></a>; Deucher, Alexander <a class="moz-txt-link-rfc2396E" href="mailto:Alexander.Deucher@amd.com"><Alexander.Deucher@amd.com></a> Cc: <a class="moz-txt-link-abbreviated" href="mailto:amd-gfx@lists.freedesktop.org">amd-gfx@lists.freedesktop.org</a>; Ding, Pixel <a class="moz-txt-link-rfc2396E" href="mailto:Pixel.Ding@amd.com"><Pixel.Ding@amd.com></a>; Jiang, Jerry (SW) <a class="moz-txt-link-rfc2396E" href="mailto:Jerry.Jiang@amd.com"><Jerry.Jiang@amd.com></a>; Li, Bingley <a class="moz-txt-link-rfc2396E" href="mailto:Bingley.Li@amd.com"><Bingley.Li@amd.com></a>; Ramirez, Alejandro <a class="moz-txt-link-rfc2396E" href="mailto:Alejandro.Ramirez@amd.com"><Alejandro.Ramirez@amd.com></a>; Filipas, Mario <a class="moz-txt-link-rfc2396E" href="mailto:Mario.Filipas@amd.com"><Mario.Filipas@amd.com></a> Subject: Re: TDR and VRAM lost handling in KMD:<o:p></o:p> </div> </div> <o:p> </o:p> <div id="divtagdefaultwrapper"> From a Mesa perspective, this almost all sounds reasonable to me.<o:p></o:p> <o:p> </o:p> On "guilty": "guilty" is a term that's used by APIs (e.g. OpenGL), so it's reasonable to use it. However, it does not make sense to mark idle contexts as "guilty" just because VRAM is lost. VRAM lost is a perfect example where the driver should report context lost to applications with the "innocent" flag for contexts that were idle at the time of reset. The only context(s) that should be reported as "guilty" (or perhaps "unknown" in some cases) are the ones that were executing at the time of reset.<o:p></o:p> <o:p> </o:p> On whether the whole context is marked as guilty from a user space perspective, it would simply be nice for user space to get consistent answers. It would be a bit odd if we could e.g. succeed in submitting an SDMA job after a GFX job was rejected. This would point in favor of marking the entire context as guilty (although that could happen lazily instead of at reset time). On the other hand, if that's too big a burden for the kernel implementation I'm sure we can live without it.<o:p></o:p> <o:p> </o:p> Cheers,<o:p></o:p> Nicolai<o:p></o:p> </div> <div class="MsoNormal" style="text-align:center" align="center"> <hr size="3" align="center" width="98%"> </div> <div id="divRplyFwdMsg"> From: Liu, Monk Sent: Wednesday, October 11, 2017 10:15:40 AM To: Koenig, Christian; Haehnle, Nicolai; Olsak, Marek; Deucher, Alexander Cc: <a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">amd-gfx@lists.freedesktop.org</a>; Ding, Pixel; Jiang, Jerry (SW); Li, Bingley; Ramirez, Alejandro; Filipas, Mario Subject: RE: TDR and VRAM lost handling in KMD: <o:p></o:p> <div> <o:p></o:p> </div> </div> <div> 1. Set its fence error status to “ETIME”,<o:p></o:p> No, as I already explained ETIME is for synchronous operation. In other words when we return ETIME from the wait IOCTL it would mean that the waiting has somehow timed out, but not the job we waited for. Please use ECANCELED as well or some other error code when we find that we need to distinct the timedout job from the canceled ones (probably a good idea, but I'm not sure).<o:p></o:p> <o:p> </o:p> [ML] I’m okay if you insist not to use ETIME<o:p></o:p> <o:p> </o:p> 1. Find the entity/ctx behind this job, and set this ctx as “guilty”<o:p></o:p> Not sure. Do we want to set the whole context as guilty or just the entity? Setting the whole contexts as guilty sounds racy to me. BTW: We should use a different name than "guilty", maybe just "bool canceled;" ?<o:p></o:p> <o:p> </o:p> [ML] I think context is better than entity, because for example if you only block entity_0 of context and allow entity_N run, that means the dependency between entities are broken (e.g. page table updates in <o:p></o:p> Sdma entity pass but gfx submit in GFX entity blocked, not make sense to me)<o:p></o:p> We’d better either block the whole context or let not… <o:p></o:p> <o:p> </o:p> <o:p> </o:p> 1. Kick out all jobs in this “guilty” ctx’s KFIFO queue, and set all their fence status to “ECANCELED”<o:p></o:p> Setting ECANCELED should be ok. But I think we should do this when we try to run the jobs and not during GPU reset.<o:p></o:p> <o:p> </o:p> [ML] without deep thought and expritment, I’m not sure the difference between them, but kick it out in gpu_reset routine is more efficient, <o:p></o:p> Otherwise you need to check context/entity guilty flag in run_job routine … and you need to it for every context/entity, I don’t see why <o:p></o:p> We don’t just kickout all of them in gpu_reset stage ….<o:p></o:p> <o:p> </o:p> <o:p> </o:p> a) Iterate over all living ctx, and set all ctx as “guilty” since VRAM lost actually ruins all VRAM contents<o:p></o:p> No, that shouldn't be done by comparing the counters. Iterating over all contexts is way to much overhead.<o:p></o:p> <o:p> </o:p> [ML] because I want to make KMS IOCTL rules clean, like they don’t need to differentiate VRAM lost or not, they only interested in if the context is guilty or not, and block<o:p></o:p> Submit for guilty ones. <o:p></o:p> <o:p> </o:p> Can you give more details of your idea? And better the detail implement in cs_submit, I want to see how you want to block submit without checking context guilty flag<o:p></o:p> <o:p> </o:p> <o:p> </o:p> a) Kick out all jobs in all ctx’s KFIFO queue, and set all their fence status to “ECANCELDED”<o:p></o:p> Yes and no, that should be done when we try to run the jobs and not during GPU reset.<o:p></o:p> <o:p> </o:p> [ML] again, kicking out them in gpu reset routine is high efficient, otherwise you need check on every job in run_job()<o:p></o:p> Besides, can you illustrate the detail implementation ?<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> Yes and no, dma_fence_get_status() is some specific handling for sync_file debugging (no idea why that made it into the common fence code). It was replaced by putting the error code directly into the fence, so just reading that one after waiting should be ok. Maybe we should fix dma_fence_get_status() to do the right thing for this?<o:p></o:p> <o:p> </o:p> [ML] yeah, that’s too confusing, the name sound really the one I want to use, we should change it…<o:p></o:p> But look into the implement, I don’t see why we cannot use it ? it also finally return the fence->error <o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Koenig, Christian Sent: Wednesday, October 11, 2017 3:21 PM To: Liu, Monk <<a href="mailto:Monk.Liu@amd.com" moz-do-not-send="true">Monk.Liu@amd.com</a>>; Haehnle, Nicolai <<a href="mailto:Nicolai.Haehnle@amd.com" moz-do-not-send="true">Nicolai.Haehnle@amd.com</a>>; Olsak, Marek <<a href="mailto:Marek.Olsak@amd.com" moz-do-not-send="true">Marek.Olsak@amd.com</a>>; Deucher, Alexander <<a href="mailto:Alexander.Deucher@amd.com" moz-do-not-send="true">Alexander.Deucher@amd.com</a>> Cc: <a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">amd-gfx@lists.freedesktop.org</a>; Ding, Pixel <<a href="mailto:Pixel.Ding@amd.com" moz-do-not-send="true">Pixel.Ding@amd.com</a>>; Jiang, Jerry (SW) <<a href="mailto:Jerry.Jiang@amd.com" moz-do-not-send="true">Jerry.Jiang@amd.com</a>>; Li, Bingley <<a href="mailto:Bingley.Li@amd.com" moz-do-not-send="true">Bingley.Li@amd.com</a>>; Ramirez, Alejandro <<a href="mailto:Alejandro.Ramirez@amd.com" moz-do-not-send="true">Alejandro.Ramirez@amd.com</a>>; Filipas, Mario <<a href="mailto:Mario.Filipas@amd.com" moz-do-not-send="true">Mario.Filipas@amd.com</a>> Subject: Re: TDR and VRAM lost handling in KMD:<o:p></o:p> </div> </div> <o:p> </o:p> <div> See inline: Am 11.10.2017 um 07:33 schrieb Liu, Monk:<o:p></o:p> </div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> Hi Christian & Nicolai,<o:p></o:p> <o:p></o:p> We need to achieve some agreements on what should MESA/UMD do and what should KMD do, please give your comments with “okay” or “No” and your idea on below items,<o:p></o:p> <o:p></o:p> ? When a job timed out (set from lockup_timeout kernel parameter), What KMD should do in TDR routine :<o:p></o:p> <o:p></o:p> 1. Update adev->gpu_reset_counter, and stop scheduler first, (gpu_reset_counter is used to force vm flush after GPU reset, out of this thread’s scope so no more discussion on it)<o:p></o:p> </blockquote> Okay. <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> 2. Set its fence error status to “ETIME”,<o:p></o:p> </blockquote> No, as I already explained ETIME is for synchronous operation. In other words when we return ETIME from the wait IOCTL it would mean that the waiting has somehow timed out, but not the job we waited for. Please use ECANCELED as well or some other error code when we find that we need to distinct the timedout job from the canceled ones (probably a good idea, but I'm not sure). <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> 3. Find the entity/ctx behind this job, and set this ctx as “guilty”<o:p></o:p> </blockquote> Not sure. Do we want to set the whole context as guilty or just the entity? Setting the whole contexts as guilty sounds racy to me. BTW: We should use a different name than "guilty", maybe just "bool canceled;" ? <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> 4. Kick out this job from scheduler’s mirror list, so this job won’t get re-scheduled to ring anymore.<o:p></o:p> </blockquote> Okay. <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> 5. Kick out all jobs in this “guilty” ctx’s KFIFO queue, and set all their fence status to “ECANCELED”<o:p></o:p> </blockquote> Setting ECANCELED should be ok. But I think we should do this when we try to run the jobs and not during GPU reset. <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> 6. Force signal all fences that get kicked out by above two steps, otherwise UMD will block forever if waiting on those fences<o:p></o:p> </blockquote> Okay. <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> 7. Do gpu reset, which is can be some callbacks to let bare-metal and SR-IOV implement with their favor style <o:p></o:p> </blockquote> Okay. <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> 8. After reset, KMD need to aware if the VRAM lost happens or not, bare-metal can implement some function to judge, while for SR-IOV I prefer to read it from GIM side (for initial version we consider it’s always VRAM lost, till GIM side change aligned)<o:p></o:p> </blockquote> Okay. <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> 9. If VRAM lost not hit, continue, otherwise:<o:p></o:p> a) Update adev->vram_lost_counter,<o:p></o:p> </blockquote> Okay. <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> b) Iterate over all living ctx, and set all ctx as “guilty” since VRAM lost actually ruins all VRAM contents<o:p></o:p> </blockquote> No, that shouldn't be done by comparing the counters. Iterating over all contexts is way to much overhead. <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> c) Kick out all jobs in all ctx’s KFIFO queue, and set all their fence status to “ECANCELDED”<o:p></o:p> </blockquote> Yes and no, that should be done when we try to run the jobs and not during GPU reset. <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> 10. Do GTT recovery and VRAM page tables/entries recovery (optional, do we need it ???)<o:p></o:p> </blockquote> Yes, that is still needed. As Nicolai explained we can't be sure that VRAM is still 100% correct even when it isn't cleared. <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> 11. Re-schedule all JOBs remains in mirror list to ring again and restart scheduler (for VRAM lost case, no JOB will re-scheduled)<o:p></o:p> </blockquote> Okay. <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <o:p></o:p> ? For cs_wait() IOCTL:<o:p></o:p> After it found fence signaled, it should check with “dma_fence_get_status” to see if there is error there,<o:p></o:p> And return the error status of fence<o:p></o:p> </blockquote> Yes and no, dma_fence_get_status() is some specific handling for sync_file debugging (no idea why that made it into the common fence code). It was replaced by putting the error code directly into the fence, so just reading that one after waiting should be ok. Maybe we should fix dma_fence_get_status() to do the right thing for this? <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <o:p></o:p> ? For cs_wait_fences() IOCTL:<o:p></o:p> Similar with above approach<o:p></o:p> <o:p></o:p> ? For cs_submit() IOCTL:<o:p></o:p> It need to check if current ctx been marked as “guilty” and return “ECANCELED” if so<o:p></o:p> <o:p></o:p> ? Introduce a new IOCTL to let UMD query vram_lost_counter:<o:p></o:p> This way, UMD can also block app from submitting, like @Nicolai mentioned, we can cache one copy of vram_lost_counter when enumerate physical device, and deny all <o:p></o:p> gl-context from submitting if the counter queried bigger than that one cached in physical device. (looks a little overkill to me, but easy to implement ) <o:p></o:p> UMD can also return error to APP when creating gl-context if found current queried vram_lost_counter bigger than that one cached in physical device.<o:p></o:p> </blockquote> Okay. Already have a patch for this, please review that one if you haven't already done so. Regards, Christian. <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <o:p></o:p> BTW: I realized that gl-context is a little different with kernel’s context. Because for kernel. BO is not related with context but only with FD, while in UMD, BO have a backend<o:p></o:p> gl-context, so block submitting in UMD layer is also needed although KMD will do its job as bottom line <o:p></o:p> <o:p></o:p> ? Basically “vram_lost_counter” is exposure by kernel to let UMD take the control of robust extension feature, it will be UMD’s call to move, KMD only deny “guilty” context from submitting<o:p></o:p> <o:p></o:p> <o:p></o:p> Need your feedback, thx<o:p></o:p> <o:p></o:p> We’d better make TDR feature landed ASAP<o:p></o:p> <o:p></o:p> BR Monk<o:p></o:p> <o:p></o:p> <o:p></o:p> <o:p></o:p> <o:p></o:p> </blockquote> <o:p> </o:p> </div> </div> </blockquote> </body> </html>