<div dir="auto"><div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 3, 2023, 14:53 André Almeida <<a href="mailto:andrealmeid@igalia.com">andrealmeid@igalia.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Em 03/05/2023 14:08, Marek Olšák escreveu: > GPU hangs are pretty common post-bringup. They are not common per user, > but if we gather all hangs from all users, we can have lots and lots of > them. > > GPU hangs are indeed not very debuggable. There are however some things > we can do: > - Identify the hanging IB by its VA (the kernel should know it) How can the kernel tell which VA range is being executed? I only found that information at mmCP_IB1_BASE_ regs, but as stated in this thread by Christian this is not reliable to be read. </blockquote></div></div><div dir="auto"> </div><div dir="auto">The kernel receives the VA and the size via the CS ioctl. When user queues are enabled, the kernel will no longer receive them.</div><div dir="auto"> </div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> > - Read and parse the IB to detect memory corruption. > - Print active waves with shader disassembly if SQ isn't hung (often > it's not). > > Determining which packet the CP is stuck on is tricky. The CP has 2 > engines (one frontend and one backend) that work on the same command > buffer. The frontend engine runs ahead, executes some packets and > forwards others to the backend engine. Only the frontend engine has the > command buffer VA somewhere. The backend engine only receives packets > from the frontend engine via a FIFO, so it might not be possible to tell > where it's stuck if it's stuck. Do they run at the same asynchronously or does the front end waits the back end to execute? </blockquote></div></div><div dir="auto"> </div><div dir="auto">They run asynchronously and should run asynchronously for performance, but they can be synchronized using a special packet (PFP_SYNC_ME).</div><div dir="auto"> </div><div dir="auto">Marek</div><div dir="auto"> </div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> > > When the gfx pipeline hangs outside of shaders, making a scandump seems > to be the only way to have a chance at finding out what's going wrong, > and only AMD-internal versions of hw can be scanned. > > Marek > > On Wed, May 3, 2023 at 11:23 AM Christian König > <<a href="mailto:ckoenig.leichtzumerken@gmail.com" target="_blank" rel="noreferrer">ckoenig.leichtzumerken@gmail.com</a> > <mailto:<a href="mailto:ckoenig.leichtzumerken@gmail.com" target="_blank" rel="noreferrer">ckoenig.leichtzumerken@gmail.com</a>>> wrote: > > Am 03.05.23 um 17:08 schrieb Felix Kuehling: > > Am 2023-05-03 um 03:59 schrieb Christian König: > >> Am 02.05.23 um 20:41 schrieb Alex Deucher: > >>> On Tue, May 2, 2023 at 11:22 AM Timur Kristóf > >>> <<a href="mailto:timur.kristof@gmail.com" target="_blank" rel="noreferrer">timur.kristof@gmail.com</a> <mailto:<a href="mailto:timur.kristof@gmail.com" target="_blank" rel="noreferrer">timur.kristof@gmail.com</a>>> wrote: > >>>> [SNIP] > >>>>>>>> In my opinion, the correct solution to those problems would be > >>>>>>>> if > >>>>>>>> the kernel could give userspace the necessary information > about > >>>>>>>> a > >>>>>>>> GPU hang before a GPU reset. > >>>>>>>> > >>>>>>> The fundamental problem here is that the kernel doesn't have > >>>>>>> that > >>>>>>> information either. We know which IB timed out and can > >>>>>>> potentially do > >>>>>>> a devcoredump when that happens, but that's it. > >>>>>> > >>>>>> Is it really not possible to know such a fundamental thing > as what > >>>>>> the > >>>>>> GPU was doing when it hung? How are we supposed to do any > kind of > >>>>>> debugging without knowing that? > >> > >> Yes, that's indeed something at least I try to figure out for years > >> as well. > >> > >> Basically there are two major problems: > >> 1. When the ASIC is hung you can't talk to the firmware engines any > >> more and most state is not exposed directly, but just through some > >> fw/hw interface. > >> Just take a look at how umr reads the shader state from the SQ. > >> When that block is hung you can't do that any more and basically > have > >> no chance at all to figure out why it's hung. > >> > >> Same for other engines, I remember once spending a week > figuring > >> out why the UVD block is hung during suspend. Turned out to be a > >> debugging nightmare because any time you touch any register of that > >> block the whole system would hang. > >> > >> 2. There are tons of things going on in a pipeline fashion or even > >> completely in parallel. For example the CP is just the beginning > of a > >> rather long pipeline which at the end produces a bunch of pixels. > >> In almost all cases I've seen you ran into a problem somewhere > >> deep in the pipeline and only very rarely at the beginning. > >> > >>>>>> > >>>>>> I wonder what AMD's Windows driver team is doing with this > problem, > >>>>>> surely they must have better tools to deal with GPU hangs? > >>>>> For better or worse, most teams internally rely on scan dumps via > >>>>> JTAG > >>>>> which sort of limits the usefulness outside of AMD, but also > gives > >>>>> you > >>>>> the exact state of the hardware when it's hung so the > hardware teams > >>>>> prefer it. > >>>>> > >>>> How does this approach scale? It's not something we can ask > users to > >>>> do, and even if all of us in the radv team had a JTAG device, we > >>>> wouldn't be able to play every game that users experience > random hangs > >>>> with. > >>> It doesn't scale or lend itself particularly well to external > >>> development, but that's the current state of affairs. > >> > >> The usual approach seems to be to reproduce a problem in a lab and > >> have a JTAG attached to give the hw guys a scan dump and they can > >> then tell you why something didn't worked as expected. > > > > That's the worst-case scenario where you're debugging HW or FW > issues. > > Those should be pretty rare post-bringup. But are there hangs caused > > by user mode driver or application bugs that are easier to debug and > > probably don't even require a GPU reset? For example most VM faults > > can be handled without hanging the GPU. Similarly, a shader in an > > endless loop should not require a full GPU reset. In the KFD compute > > case, that's still preemptible and the offending process can be > killed > > with Ctrl-C or debugged with rocm-gdb. > > We also have infinite loop in shader abort for gfx and page faults are > pretty rare with OpenGL (a bit more often with Vulkan) and can be > handled gracefully on modern hw (they just spam the logs). > > The majority of the problems is unfortunately that we really get hard > hangs because of some hw issues. That can be caused by unlucky timing, > power management or doing things in an order the hw doesn't expected. > > Regards, > Christian. > > > > > It's more complicated for graphics because of the more complex > > pipeline and the lack of CWSR. But it should still be possible to do > > some debugging without JTAG if the problem is in SW and not HW or > FW. > > It's probably worth improving that debugability without getting > > hung-up on the worst case. > > > > Maybe user mode graphics queues will offer a better way of > recovering > > from these kinds of bugs, if the graphics pipeline can be unstuck > > without a GPU reset, just by killing the offending user mode queue. > > > > Regards, > > Felix > > > > > >> > >> And yes that absolutely doesn't scale. > >> > >> Christian. > >> > >>> > >>> Alex > >> > </blockquote></div></div></div>