GPU hang & mmu fault on GC7000

[Simon Zeni]

Hi,

I'm experiencing GPU hang with on a i.MX8 Quad Max (GC7000 rev 6009) with
linux-fslc [1] on the 6.1-2.2.x-imx branch. I've reproduced the issue on the
5.15-2.2.x-imx branch and I have not yet sorted out the 6.6.x+fslc with this
device.

Both issues are reproducible with `glmark-es2-drm` using the benchmarks
`terrain` and the `desktop:effect=shadow:windows=4`. The latter has to 
be run
with `--run-forever` to produce the hang.

 > [  774.318157] etnaviv-gpu 53100000.gpu: GPU not yet idle, mask: 
0x7fffbf76
 > [  774.830151] etnaviv-gpu 53100000.gpu: GPU not yet idle, mask: 
0x7fffbf76
 > [  775.342025] etnaviv-gpu 53100000.gpu: recover hung GPU!

The MMU fault only occurs with the `terrain` benchmark`.

 > [  775.354701] etnaviv-gpu 53100000.gpu: MMU fault status 0x00000002
 > [  775.360796] etnaviv-gpu 53100000.gpu: MMU 0 fault addr 0x00000000

I have collected devcoredumps from both benchmarks, they both share the same
idle state, dma debug state and dma address:

 > 00000004 = 7fffbf76 Idle: FE- DE+ PE+ SH- PA+ SE+ RA+ TX- VG+ IM+ FP+ TS+
 > 00000660 = 00000001 Cmd: [dec DMA: idle Fetch: idle] Req idle Cal idle
 > 00000664 = 00000905 Command DMA address

Both of them also have the line `Checking MMU entries... failed` before 
listing
the buffer offsets, is that expected?

I have searched the IRC logs (thanks for keeping this channel logged), 
and if I
get this correctly this address is an offset in the mmu.bin file 
generated by
`viv-unpack` [2]. The result in both dumps seems inconclusive.

 > 00000900  02 00 00 00 02 00 00 00  02 00 00 00 02 00 00 00 
|................|

We figured out recently that the DMA address might be corrupted, dumping it
first [3] gives a different result. Am I going in the right direction to
understand, debug and fix this (those?) issues? What more can I do, and 
where
should I look?

Thanks,

Simon

[1]: https://github.com/Freescale/linux-fslc
[2]: https://github.com/etnaviv/etna-gpu-tools/blob/master/dump/viv-unpack.c
[3]: https://lists.freedesktop.org/archives/etnaviv/2024-May/004744.html