[BUG 4.17] etnaviv-gpu f1840000.gpu: recover hung GPU!
Russell King - ARM Linux
linux at armlinux.org.uk
Wed Jul 11 20:13:44 UTC 2018
On Tue, Jun 19, 2018 at 02:28:46PM +0200, Lucas Stach wrote:
> See attached patch (which I apparently forgot to send out). The DRM GPU
> scheduler has some tracepoints, which might be helpful. The attached
> patch adds a drm_sched_job_run tracepoint when a job is queued in the
> hardware ring. Together with the existing drm_sched_process_job, this
> should get you an idea how long a job takes to process. Note that at
> any time up to 4 jobs are allowed in the hardware queue, so you need to
> match up the end times.
Here's the trace:
Xorg 1009 [000] 276.593018: gpu_scheduler:drm_sched_job: entity=0xee9c3e00, id=144, fence=0xeb849580, ring=ef16bd00, job count:0, hw job count:0
gnome-software 1384 [000] 276.673634: gpu_scheduler:drm_sched_process_job: fence=0xeb849580 signaled
Xorg 1009 [000] 278.756538: gpu_scheduler:drm_sched_job: entity=0xee9c3e00, id=145, fence=0xecaaef40, ring=ef16bd00, job count:0, hw job count:0
Xorg 1009 [000] 279.130196: gpu_scheduler:drm_sched_process_job: fence=0xecaaef40 signaled
Xorg 1009 [000] 279.856129: gpu_scheduler:drm_sched_job: entity=0xee9c3e00, id=146, fence=0xecbdd7c0, ring=ef16bd00, job count:0, hw job count:0
Xorg 1009 [000] 279.856542: gpu_scheduler:drm_sched_job: entity=0xee9c3e00, id=147, fence=0xeb849700, ring=ef16bd00, job count:0, hw job count:1
kswapd0 26 [000] 281.532808: gpu_scheduler:drm_sched_process_job: fence=0xecbdd7c0 signaled
gnome-software 1384 [000] 281.560666: gpu_scheduler:drm_sched_process_job: fence=0xeb849700 signaled
Xorg 1009 [000] 282.554920: gpu_scheduler:drm_sched_job: entity=0xee9c3e00, id=148, fence=0xecad3a00, ring=ef16bd00, job count:0, hw job count:0
Xorg 1009 [000] 283.956216: gpu_scheduler:drm_sched_job: entity=0xee9c3e00, id=149, fence=0xeb849700, ring=ef16bd00, job count:0, hw job count:1
Xorg 1009 [000] 284.149557: gpu_scheduler:drm_sched_process_job: fence=0xecad3a00 signaled
Xorg 1009 [000] 285.465863: gpu_scheduler:drm_sched_process_job: fence=0xeb849700 signaled
Xorg 1009 [000] 286.796692: gpu_scheduler:drm_sched_job: entity=0xee9c3e00, id=150, fence=0xecbdd340, ring=ef16bd00, job count:0, hw job count:0
Xorg 1009 [000] 288.549507: gpu_scheduler:drm_sched_job: entity=0xee9c3e00, id=151, fence=0xecbdd280, ring=ef16bd00, job count:0, hw job count:1
Xorg 1009 [000] 288.856277: gpu_scheduler:drm_sched_job: entity=0xee9c3e00, id=152, fence=0xecaaee80, ring=ef16bd00, job count:0, hw job count:2
Xorg 1009 [000] 288.856713: gpu_scheduler:drm_sched_job: entity=0xee9c3e00, id=153, fence=0xecad3040, ring=ef16bd00, job count:0, hw job count:3
gnome-software 1384 [000] 288.958764: gpu_scheduler:drm_sched_process_job: fence=0xecbdd340 signaled
Xorg 1009 [000] 290.052842: gpu_scheduler:drm_sched_job: entity=0xee9c3e00, id=154, fence=0xecbdde80, ring=ef16bd00, job count:0, hw job count:3
Xorg 1009 [000] 291.682222: gpu_scheduler:drm_sched_job: entity=0xee9c3e00, id=155, fence=0xecaaeb80, ring=ef16bd00, job count:0, hw job count:4
gnome-software 1384 [000] 291.760295: gpu_scheduler:drm_sched_process_job: fence=0xecbdd280 signaled
gnome-software 1384 [000] 292.018437: gpu_scheduler:drm_sched_process_job: fence=0xecaaee80 signaled
Xorg 1009 [000] 292.046721: gpu_scheduler:drm_sched_process_job: fence=0xecad3040 signaled
Xorg 1009 [000] 292.819475: gpu_scheduler:drm_sched_process_job: fence=0xecbdde80 signaled
Xorg 1009 [000] 293.149829: gpu_scheduler:drm_sched_job: entity=0xee9c3e00, id=156, fence=0xecad3f40, ring=ef16bd00, job count:0, hw job count:1
gnome-software 1384 [000] 293.190607: gpu_scheduler:drm_sched_process_job: fence=0xecaaeb80 signaled
gnome-software 1384 [000] 293.190720: gpu_scheduler:drm_sched_process_job: fence=0xecad3f40 signaled
Xorg 1009 [000] 293.463414: gpu_scheduler:drm_sched_job: entity=0xee9c3e00, id=157, fence=0xecbddd00, ring=ef16bd00, job count:0, hw job count:0
Xorg 1009 [000] 293.463629: gpu_scheduler:drm_sched_process_job: fence=0xecbddd00 signaled
So:
Job Start End Time on queue:
144 276.593018 276.673634 80ms
145 278.756538 279.130196 374ms
146 279.856129 281.532808 1.7s
147 279.856542 281.560666 1.7s (overlaps 146)
148 282.554920 284.149557 1.6s
149 283.956216 285.465863 1.5s (overlaps 148)
150 286.796692 288.958764 2.2s
151 288.549507 291.760295 3.2s (overlaps 150)
152 288.856277 292.018437 3.2s (overlaps 150-151)
153 288.856713 292.046721 3.2s (overlaps 150-152)
154 290.052842 292.819475 2.8s (overlaps 151-153)
155 291.682222 293.190607 1.5s (overlaps 151-154)
156 293.149829 293.190720 40ms (overlaps 155)
157 293.463414 293.463629 215us
The time for jobs 147, 149, 151-155 are misleading because they are
delayed by the preceeding job. Correcting for that,
Time executing:
147 28ms
149 1.3s
151 2.8s
152 250ms
153 28ms
154 780ms
155 370ms
So, sorted by time apparently executing:
151 2.8s
150 286.796692 288.958764 2.2s
146 279.856129 281.532808 1.7s
148 282.554920 284.149557 1.6s
149 1.3s
154 780ms
145 278.756538 279.130196 374ms
155 370ms
152 250ms
144 276.593018 276.673634 80ms
156 293.149829 293.190720 40ms
153 28ms
147 28ms
157 293.463414 293.463629 215us
Of course, from the perf dump, we don't know what the GPU was being
asked to do.
I've applied your forward-progress patch, and with the 5s timeout
reverted, it seems to "behave" without causing a timeout.
--
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 13.8Mbps down 630kbps up
According to speedtest.net: 13Mbps down 490kbps up
More information about the etnaviv
mailing list