Help debug amdgpu faults

[Alex Deucher]

On Tue, Nov 22, 2022 at 6:53 AM Robert Beckett
<bob.beckett at collabora.com> wrote:
>
> Hi,
>
>
> does anyone know any documentation, or can provide advice on debugging amdgpu fault reports?

This is a GPU page fault so it refers the the GPU virtual address
space of the application .  Each process (well fd really), gets its
own GPU virtual address space into which system memory, system mmio
space, or vram can be mapped.  The user mode drivers control their GPU
virtual address space.

>
>
> e.g:
>
> Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:8 vmid:1 pasid:32769, for process vkcube pid 999 thread vkcube pid 999)

This is the process that caused the fault.

> Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu:   in page starting at address 0x0000800100700000 from client 0x1b (UTCL2)

This is the virtual address that faulted.

> Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00101A10
> Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu:          Faulty UTCL2 client ID: SDMA0 (0xd)

The fault came from the SDMA engine.

> Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu:          MORE_FAULTS: 0x0
> Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu:          WALKER_ERROR: 0x0
> Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu:          PERMISSION_FAULTS: 0x1

The page was not marked as valid in the GPU page table.

> Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu:          MAPPING_ERROR: 0x0
> Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu:          RW: 0x0

SDMA attempted to read an invalid page.

>
>
>
> see https://gitlab.freedesktop.org/drm/amd/-/issues/2267 for more context.
>
> We have a complicated setup involving rendering then blitting to virtio-gpu exported dmabufs, with plenty of hacks in the mesa and xwayland stacks, so we are considering this our issue to debug, and not an issue with the driver at this point.
> However, having debugged all the interesting parts leading to these faults, I am unable to decode the fault report to correlate to a buffer.
>
> in the fault report, what address space is the address from?
> given that the fault handler shifts the reported addres up by 12, I assume it is a 4K pfn which makes me assume a physical address is this correct?
> if so, is that a vram pa or a host system memory pa?
> is there any documentation for the other fields reported like the fault status etc?

See the comments above.  There is some kernel doc as well:
https://docs.kernel.org/gpu/amdgpu/driver-core.html#amdgpu-virtual-memory

>
> I'd appreciate any advice you could give to help us debug further.

Some operation you are doing in the user mode driver is reading an
invalid page.  Possibly reading past the end of a buffer or something
mis-aligned.  Compare the faulting GPU address to the GPU virtual
address space in the application and you should be able to track down
what is happening.

Alex

>
> Thanks
>
> Bob
>