<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 11 Mar 2021 at 21:00, Daniel Gomez <<a href="mailto:daniel@qtec.com" target="_blank">daniel@qtec.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Thu, 11 Mar 2021 at 17:10, Alex Deucher <<a href="mailto:alexdeucher@gmail.com" target="_blank">alexdeucher@gmail.com</a>> wrote:<br> ><br> > On Thu, Mar 11, 2021 at 10:02 AM Alexandre Desnoyers <<a href="mailto:alex@qtec.com" target="_blank">alex@qtec.com</a>> wrote:<br> > ><br> > > On Thu, Mar 11, 2021 at 2:49 PM Daniel Gomez <<a href="mailto:daniel@qtec.com" target="_blank">daniel@qtec.com</a>> wrote:<br> > > ><br> > > > On Thu, 11 Mar 2021 at 10:09, Daniel Gomez <<a href="mailto:daniel@qtec.com" target="_blank">daniel@qtec.com</a>> wrote:<br> > > > ><br> > > > > On Wed, 10 Mar 2021 at 18:06, Alex Deucher <<a href="mailto:alexdeucher@gmail.com" target="_blank">alexdeucher@gmail.com</a>> wrote:<br> > > > > ><br> > > > > > On Wed, Mar 10, 2021 at 11:37 AM Daniel Gomez <<a href="mailto:daniel@qtec.com" target="_blank">daniel@qtec.com</a>> wrote:<br> > > > > > ><br> > > > > > > Disabling GFXOFF via the quirk list fixes a hardware lockup in<br> > > > > > > Ryzen V1605B, RAVEN 0x1002:0x15DD rev 0x83.<br> > > > > > ><br> > > > > > > Signed-off-by: Daniel Gomez <<a href="mailto:daniel@qtec.com" target="_blank">daniel@qtec.com</a>><br> > > > > > > ---<br> > > > > > ><br> > > > > > > This patch is a continuation of the work here:<br> > > > > > > <a href="https://lkml.org/lkml/2021/2/3/122" rel="noreferrer" target="_blank">https://lkml.org/lkml/2021/2/3/122</a> where a hardware lockup was discussed and<br> > > > > > > a dma_fence deadlock was provoke as a side effect. To reproduce the issue<br> > > > > > > please refer to the above link.<br> > > > > > ><br> > > > > > > The hardware lockup was introduced in 5.6-rc1 for our particular revision as it<br> > > > > > > wasn't part of the new blacklist. Before that, in kernel v5.5, this hardware was<br> > > > > > > working fine without any hardware lock because the GFXOFF was actually disabled<br> > > > > > > by the if condition for the CHIP_RAVEN case. So this patch, adds the 'Radeon<br> > > > > > > Vega Mobile Series [1002:15dd] (rev 83)' to the blacklist to disable the GFXOFF.<br> > > > > > ><br> > > > > > > But besides the fix, I'd like to ask from where this revision comes from. Is it<br> > > > > > > an ASIC revision or is it hardcoded in the VBIOS from our vendor? From what I<br> > > > > > > can see, it comes from the ASIC and I wonder if somehow we can get an APU in the<br> > > > > > > future, 'not blacklisted', with the same problem. Then, should this table only<br> > > > > > > filter for the vendor and device and not the revision? Do you know if there are<br> > > > > > > any revisions for the 1002:15dd validated, tested and functional?<br> > > > > ><br> > > > > > The pci revision id (RID) is used to specify the specific SKU within a<br> > > > > > family. GFXOFF is supposed to be working on all raven variants. It<br> > > > > > was tested and functional on all reference platforms and any OEM<br> > > > > > platforms that launched with Linux support. There are a lot of<br> > > > > > dependencies on sbios in the early raven variants (0x15dd), so it's<br> > > > > > likely more of a specific platform issue, but there is not a good way<br> > > > > > to detect this so we use the DID/SSID/RID as a proxy. The newer raven<br> > > > > > variants (0x15d8) have much better GFXOFF support since they all<br> > > > > > shipped with newer firmware and sbios.<br> > > > ><br> > > > > We took one of the first reference platform boards to design our<br> > > > > custom board based on the V1605B and I assume it has one of the early 'unstable'<br> > > > > raven variants with RID 0x83. Also, as OEM we are in control of the bios<br> > > > > (provided by insyde) but I wasn't sure about the RID so, thanks for the<br> > > > > clarification. Is there anything we can do with the bios to have the GFXOFF<br> > > > > enabled and 'stable' for this particular revision? Otherwise we'd need to add<br> > > > > the 0x83 RID to the table. Also, there is an extra ']' in the patch<br> > > > > subject. Sorry<br> > > > > for that. Would you need a new patch in case you accept it with the ']' removed?<br> > > > ><br> > > > > Good to hear that the newer raven versions have better GFXOFF support.<br> > > ><br> > > > Adding Alex Desnoyer to the loop as he is the electronic/hardware and<br> > > > bios responsible so, he can<br> > > > provide more information about this.<br> > ><br> > > Hello everyone,<br> > ><br> > > We, Qtechnology, are the OEM of the hardware platform where we<br> > > originally discovered the bug. Our platform is based on the AMD<br> > > Dibbler V-1000 reference design, with the latest Insyde BIOS release<br> > > available for the (now unsupported) Dibbler platform. We have the<br> > > Insyde BIOS source code internally, so we can make some modifications<br> > > as needed.<br> > ><br> > > The last test that Daniel and myself performed was on a standard<br> > > Dibbler PCB rev.B1 motherboard (NOT our platform), and using the<br> > > corresponding latest AMD released BIOS "RDB1109GA". As Daniel wrote,<br> > > the hardware lockup can be reproduced on the Dibbler, even if it has a<br> > > different RID that our V1605B APU.<br> > ><br> > > We also have a Neousys Technology POC-515 embedded computer (V-1000,<br> > > V1605B) in our office. The Neousys PC also uses Insyde BIOS. This<br> > > computer is also locking-up in the test.<br> > > <a href="https://www.neousys-tech.com/en/product/application/rugged-embedded/poc-500-amd-ryzen-ultra-compact-embedded-computer" rel="noreferrer" target="_blank">https://www.neousys-tech.com/en/product/application/rugged-embedded/poc-500-amd-ryzen-ultra-compact-embedded-computer</a><br> > ><br> > ><br> > > Digging into the BIOS source code, the only reference to GFXOFF is in<br> > > the SMU and PSP firmware release notes, where some bug fixes have been<br> > > mentioned for previous SMU/PSP releases. After a quick "git grep -i<br> > > gfx | grep -i off", there seems to be no mention of GFXOFF in the<br> > > Insyde UEFI (inluding AMD PI) code base. I would appreciate any<br> > > information regarding BIOS modification needed to make the GFXOFF<br> > > feature stable. As you (Alex Deucher) mentionned, it should be<br> > > functional on all AMD Raven reference platforms.<br> > ><br> ><br> > It's handled by the firmwares carried by the sbios. I'm not sure what<br> > versions off hand. Probably want to make sure you have the latest<br> > ones. Do you have an AMD partner contact? It might be best to bring<br> > this up with them.<br> I'm sure we were using the latest but let us double-check with our<br> AMD partner and insyde just in case.<br> ><br> > Regarding the issues you are seeing is this a general issue with all<br> > workloads that use the GFX shader cores? Or just specific workloads?<br> > If it's just compute workloads, you might try this patch. It may fix<br> > the issue for you.<br> Thanks Alex for the patch. I think it's kind of a general issue with all the<br> workloads but the way we've been able to reproduce it was with the<br> MatrixMultiplication test (from AMD) and clinfo. With the patch, I'm<br> still able to reproduce the problem in our custom board. I'll check it<br> tomorrow on the dribbler.<br></blockquote><div>Quick update on this:</div><div>I've tested the patch on the dribbler and the hang also occurs. What</div><div>we are now trying to figure out if we actually have the latest bios</div><div>available for the V1605B (we think we do) but this might take a while (already reported</div><div>to the AMD partner).<br></div><div>To add more info about our current bios:</div><div>Dibbler: <br></div><div>- Version: RDB1109GA</div><div>- AGESA version: RavenPi-FP5-AM4 1.1.0.9</div><div>- VBIOS 0113</div><div>QT5222 (V1605B with Radeon Vega Gfx 8):</div><div>- Version: 0.0.29<div>- AGESA version: RavenPi-FP5-AM4 1.1.0.A</div><div>- VBIOS 0115</div><div><br></div><div>Also, in the release notes for the latest amdgpu release available in the amd support page</div><div>says: "Disable the GFXOFF for OpenCL(ROCm 3.8) usage as below in the grub.</div><div>Workaround: add “amdgpu.ppfeaturemask=0xffff3fff”in the grub." which confirms the issue.<br></div><div><a href="https://drivers.amd.com/relnotes/linux_driver_2020.40_release_notes.pdf">https://drivers.amd.com/relnotes/linux_driver_2020.40_release_notes.pdf</a></div><div><br></div>My final question would be if we can/should add the patch as there are other raven</div><div>devices already added with different revision 0x1002:0x15dd {0xc8,0xc6} or we <br></div><div>should wait forAMD-partner confirmation to know if the bios/raven is buggy or not.</div><div><div><br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> ><br> > Alex<br> ><br> ><br> > ><br> > > Regards,<br> > ><br> > > Alexandre Desnoyers<br> > ><br> > ><br> > > ><br> > > > I've now done a test on the reference platform (dibbler) with the<br> > > > latest bios available<br> > > > and the hw lockup can be also reproduced with the same steps.<br> > > ><br> > > > For reference, I'm using mainline kernel 5.12-rc2.<br> > > ><br> > > > [ 5.938544] [drm] initializing kernel modesetting (RAVEN<br> > > > 0x1002:0x15DD 0x1002:0x15DD 0xC1).<br> > > > [ 5.939942] amdgpu: ATOM BIOS: 113-RAVEN-11<br> > > ><br> > > > As in the previous cases, the clocks go to 100% of usage when the hang occurs.<br> > > ><br> > > > However, when the gpu hangs, dmesg output displays the following:<br> > > ><br> > > > [ 1568.279847] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx<br> > > > timeout, signaled seq=188, emitted seq=191<br> > > > [ 1568.434084] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process<br> > > > information: process Xorg pid 311 thread Xorg:cs0 pid 312<br> > > > [ 1568.279847] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx<br> > > > timeout, signaled seq=188, emitted seq=191<br> > > > [ 1568.434084] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process<br> > > > information: process Xorg pid 311 thread Xorg:cs0 pid 312<br> > > > [ 1568.507000] amdgpu 0000:01:00.0: amdgpu: GPU reset begin!<br> > > > [ 1628.491882] rcu: INFO: rcu_sched self-detected stall on CPU<br> > > > [ 1628.491882] rcu: 3-...!: (665 ticks this GP)<br> > > > idle=f9a/1/0x4000000000000000 softirq=188533/188533 fqs=15<br> > > > [ 1628.491882] rcu: rcu_sched kthread timer wakeup didn't happen for<br> > > > 58497 jiffies! g726761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402<br> > > > [ 1628.491882] rcu: Possible timer handling issue on cpu=2<br> > > > timer-softirq=55225<br> > > > [ 1628.491882] rcu: rcu_sched kthread starved for 58500 jiffies!<br> > > > g726761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=2<br> > > > [ 1628.491882] rcu: Unless rcu_sched kthread gets sufficient CPU<br> > > > time, OOM is now expected behavior.<br> > > > [ 1628.491882] rcu: RCU grace-period kthread stack dump:<br> > > > [ 1628.491882] rcu: Stack dump where RCU GP kthread last ran:<br> > > > [ 1808.518445] rcu: INFO: rcu_sched self-detected stall on CPU<br> > > > [ 1808.518445] rcu: 3-...!: (2643 ticks this GP)<br> > > > idle=f9a/1/0x4000000000000000 softirq=188533/188533 fqs=15<br> > > > [ 1808.518445] rcu: rcu_sched kthread starved for 238526 jiffies!<br> > > > g726761 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2<br> > > > [ 1808.518445] rcu: Unless rcu_sched kthread gets sufficient CPU<br> > > > time, OOM is now expected behavior.<br> > > > [ 1808.518445] rcu: RCU grace-period kthread stack dump:<br> > > > [ 1808.518445] rcu: Stack dump where RCU GP kthread last ran:<br> > > ><br> > > > ><br> > > > > Daniel<br> > > > ><br> > > > > ><br> > > > > > Alex<br> > > > > ><br> > > > > ><br> > > > > > ><br> > > > > > > Logs:<br> > > > > > > [ 27.708348] [drm] initializing kernel modesetting (RAVEN<br> > > > > > > 0x1002:0x15DD 0x1002:0x15DD 0x83).<br> > > > > > > [ 27.789156] amdgpu: ATOM BIOS: 113-RAVEN-115<br> > > > > > ><br> > > > > > > Thanks in advance,<br> > > > > > > Daniel<br> > > > > > ><br> > > > > > > drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 2 ++<br> > > > > > > 1 file changed, 2 insertions(+)<br> > > > > > ><br> > > > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c<br> > > > > > > index 65db88bb6cbc..319d4b99aec8 100644<br> > > > > > > --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c<br> > > > > > > +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c<br> > > > > > > @@ -1243,6 +1243,8 @@ static const struct amdgpu_gfxoff_quirk amdgpu_gfxoff_quirk_list[] = {<br> > > > > > > { 0x1002, 0x15dd, 0x103c, 0x83e7, 0xd3 },<br> > > > > > > /* GFXOFF is unstable on C6 parts with a VBIOS 113-RAVEN-114 */<br> > > > > > > { 0x1002, 0x15dd, 0x1002, 0x15dd, 0xc6 },<br> > > > > > > + /* GFXOFF provokes a hw lockup on 83 parts with a VBIOS 113-RAVEN-115 */<br> > > > > > > + { 0x1002, 0x15dd, 0x1002, 0x15dd, 0x83 },<br> > > > > > > { 0, 0, 0, 0, 0 },<br> > > > > > > };<br> > > > > > ><br> > > > > > > --<br> > > > > > > 2.30.1<br> > > > > > ><br> > > > > > > _______________________________________________<br> > > > > > > dri-devel mailing list<br> > > > > > > <a href="mailto:dri-devel@lists.freedesktop.org" target="_blank">dri-devel@lists.freedesktop.org</a><br> > > > > > > <a href="https://lists.freedesktop.org/mailman/listinfo/dri-devel" rel="noreferrer" target="_blank">https://lists.freedesktop.org/mailman/listinfo/dri-devel</a><br> </blockquote></div></div>