<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
Hi Timur,<br>
<br>
<div class="moz-cite-prefix">Am 02.05.23 um 11:12 schrieb Timur
Kristóf:<br>
</div>
<blockquote type="cite" cite="mid:CAFF-SiV0=WNmRW-D9uYUuj68Zq0APxtGLya9KR6FfZ7v0Zf2RQ@mail.gmail.com">
<div dir="auto">
<div>Hi Christian,<br>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Christian König <<a href="mailto:christian.koenig@amd.com" moz-do-not-send="true" class="moz-txt-link-freetext">christian.koenig@amd.com</a>>
ezt írta (időpont: 2023. máj. 2., Ke 9:59):<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Am
02.05.23 um 03:26 schrieb André Almeida:<br>
> Em 01/05/2023 16:24, Alex Deucher escreveu:<br>
>> On Mon, May 1, 2023 at 2:58 PM André Almeida <<a href="mailto:andrealmeid@igalia.com" target="_blank" rel="noreferrer" moz-do-not-send="true" class="moz-txt-link-freetext">andrealmeid@igalia.com</a>>
<br>
>> wrote:<br>
>>><br>
>>> I know that devcoredump is also used for this
kind of information, <br>
>>> but I believe<br>
>>> that using an IOCTL is better for interfacing
Mesa + Linux rather <br>
>>> than parsing<br>
>>> a file that its contents are subjected to be
changed.<br>
>><br>
>> Can you elaborate a bit on that? Isn't the whole
point of devcoredump<br>
>> to store this sort of information?<br>
>><br>
><br>
> I think that devcoredump is something that you could
use to submit to <br>
> a bug report as it is, and then people can read/parse
as they want, <br>
> not as an interface to be read by Mesa... I'm not
sure that it's <br>
> something that I would call an API. But I might be
wrong, if you know <br>
> something that uses that as an API please share.<br>
><br>
> Anyway, relying on that for Mesa would mean that we
would need to <br>
> ensure stability for the file content and format,
making it less <br>
> flexible to modify in the future and probe to bugs,
while the IOCTL is <br>
> well defined and extensible. Maybe the dump from Mesa
+ devcoredump <br>
> could be complementary information to a bug report.<br>
<br>
Neither using an IOCTL nor devcoredump is a good approach
for this since <br>
the values read from the hw register are completely
unreliable. They <br>
could not be available because of GFXOFF or they could be
overwritten or <br>
not even updated by the CP in the first place because of a
hang etc....<br>
<br>
If you want to track progress inside an IB what you do
instead is to <br>
insert intermediate fence write commands into the IB. E.g.
something <br>
like write value X to location Y when this executes.<br>
<br>
This way you can not only track how far the IB processed,
but also in <br>
which stages of processing we where when the hang
occurred. E.g. End of <br>
Pipe, End of Shaders, specific shader stages etc...<br>
</blockquote>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Currently our biggest challenge in the userspace
driver is debugging "random" GPU hangs. We have many dozens of
bug reports from users which are like: "play the game for X
hours and it will eventually hang the GPU". With the currently
available tools, it is impossible for us to tackle these
issues. André's proposal would be a step in improving this
situation.</div>
<div dir="auto"><br>
</div>
<div dir="auto">We already do something like what you suggest,
but there are multiple problems with that approach:</div>
<div dir="auto"><br>
</div>
<div dir="auto">1. we can only submit 1 command buffer at a time
because we won't know which IB hanged</div>
<div dir="auto">2. we can't use chaining because we don't know
where in the IB it hanged</div>
<div dir="auto">3. it needs userspace to insert (a lot of) extra
commands such as extra synchronization and memory writes</div>
<div dir="auto">4. It doesn't work when GPU recovery is enabled
because the information is already gone when we detect the
hang<br>
</div>
</div>
</blockquote>
<br>
You can still submit multiple IBs and even chain them. All you need
to do is to insert into each IB commands which write to an extra
memory location with the IB executed and the position inside the IB.<br>
<br>
The write data command allows to write as many dw as you want (up to
multiple kb). The only potential problem is when you submit the same
IB multiple times.<br>
<br>
And yes that is of course quite some extra overhead, but I think
that should be manageable.<br>
<br>
<blockquote type="cite" cite="mid:CAFF-SiV0=WNmRW-D9uYUuj68Zq0APxtGLya9KR6FfZ7v0Zf2RQ@mail.gmail.com">
<div dir="auto">
<div dir="auto"><br>
</div>
<div dir="auto">Consequences:</div>
<div dir="auto"><br>
</div>
<div dir="auto">A. It has a huge perf impact, so we can't enable
it always</div>
<div dir="auto">B. Thanks to the extra synchronization, some
issues can't be reproduced when this kind of debugging is
enabled</div>
<div dir="auto">C. We have to ask users to disable GPU recovery
to collect logs for us</div>
<div dir="auto"><br>
</div>
<div dir="auto">In my opinion, the correct solution to those
problems would be if the kernel could give userspace the
necessary information about a GPU hang before a GPU reset.</div>
</div>
</blockquote>
<br>
The fundamental problem here is that the kernel doesn't have that
information either. We know which IB timed out and can potentially
do a devcoredump when that happens, but that's it.<br>
<br>
The devcoredump can contain register values and memory locations,
but since the ASIC is hung reading back stuff like the CP or power
management state is dangerous. Alex already noted as well that we
potentially need to disable GFXOFF to be able to read back register
values.<br>
<br>
<blockquote type="cite" cite="mid:CAFF-SiV0=WNmRW-D9uYUuj68Zq0APxtGLya9KR6FfZ7v0Zf2RQ@mail.gmail.com">
<div dir="auto">
<div dir="auto"> To avoid the massive peformance cost, it would
be best if we could know which IB hung and what were the
commands being executed when it hung (perhaps pointers to the
VA of the commands), along with which shaders were in flight
(perhaps pointers to the VA of the shader binaries).</div>
</div>
</blockquote>
<blockquote type="cite" cite="mid:CAFF-SiV0=WNmRW-D9uYUuj68Zq0APxtGLya9KR6FfZ7v0Zf2RQ@mail.gmail.com">
<div dir="auto">
<div dir="auto"><br>
</div>
<div dir="auto">If such an interface could be created, that
would mean we could easily query this information and create
useful logs of GPU hangs without much userspace overhead and
without requiring the user to disable GPU resets etc.</div>
</div>
</blockquote>
<br>
Disabling GPU reset is actually not something you can reliably do
either. If the core memory management runs OOM it can chose to wait
for the GPU reset to finish and when that timeout is infinite the
system just hard hangs.<br>
<br>
That's also the reason why you can't wait for userspace to read back
state or something like that.<br>
<br>
<blockquote type="cite" cite="mid:CAFF-SiV0=WNmRW-D9uYUuj68Zq0APxtGLya9KR6FfZ7v0Zf2RQ@mail.gmail.com">
<div dir="auto">
<div dir="auto"><br>
</div>
<div dir="auto">If it's not possible to do this, we'd appreciate
some suggestions on how to properly solve this without the
massive performance cost and without requiring the user to
disable GPU recovery.</div>
</div>
</blockquote>
<br>
The most doable option I can see is to say instead of resetting the
GPU we tell the OS the GPU was hot unplugged and disable system
memory access for the ASIC to prevent random memory corruption.<br>
<br>
This way you can investigate the GPU state with tools like umr and
only after that is done we do a hot add event and start over from
scratch.<br>
<br>
The game will obviously crash and needs to be restarted, but that is
still better than a full system crash.<br>
<br>
<blockquote type="cite" cite="mid:CAFF-SiV0=WNmRW-D9uYUuj68Zq0APxtGLya9KR6FfZ7v0Zf2RQ@mail.gmail.com">
<div dir="auto">
<div dir="auto"><br>
</div>
<div dir="auto">Side note, it is also extremely difficult to
even determine whether the problem is in userspace or the
kernel. While kernel developers usually dismiss all GPU hangs
as userspace problems, we've seen many issues where the
problem was in the kernel (eg. bugs where wrong voltages were
set, etc.) - any idea for tackling those kind of issues is
also welcome.<br>
</div>
</div>
</blockquote>
<br>
No idea either. If I had a better idea how to triage things my live
would be much simpler as well :)<br>
<br>
Regards,<br>
Christian.<br>
<br>
<blockquote type="cite" cite="mid:CAFF-SiV0=WNmRW-D9uYUuj68Zq0APxtGLya9KR6FfZ7v0Zf2RQ@mail.gmail.com">
<div dir="auto">
<div dir="auto"><br>
</div>
<div dir="auto">Thanks & best regards,</div>
<div dir="auto">Timur</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>