[Mesa-dev] [PATCH 00/22] RFC: Batchbuffer Logger for Intel GPU

[Chris Wilson]

Quoting Rogovin, Kevin (2017-09-27 07:53:29)
> Hi,
> 
>  Right now the way the thing works is that it walks the batchbuffer just after the kernel returns from the ioctl and updates its internal view of the GPU state as it walks and emits to the log file the data. The log on a single batchbuffer is (essentially) just a list of call ID's from the apitrace together of "where in the batchbuffer" that call started. 
> 
>  I confess that I had not realized the potential application for using something like this to help diagnose GPU hangs! I think it is a really good idea. What I could do is the following (and it is not terribly hard to do):
> 
>    1. -BEFORE- issuing the ioctl, the logger walks just the api markers in the log of the batchbuffer, and makes a new GEM BO filled with apitrace data (call ID, and maybe GL function data) and modify the ioctl to have an extra buffer.

Yes. With the current intel_batchbuffer.c this should be relatively easy
(I suggest you limit yourself to recent kernels for that
simplification);
see EXEC_BATCH_FIRST and remember to mark the trace bo as
EXEC_OBJECT_CAPTURE.
 
>   2. -AFTER- the ioctl returns, emit the log data (as now) and delete the GEM BO; In order to read the GPU state more accurately I need to walk the log and update the GPU state after the ioctl (mostly out of paranoia for values copied from BO's to pipeline registers).

Up to, but my paranoia goes the other way. Once the ioctl returns the hw
is indeed using that memory, so I have less trust of it. If you need to
tie the relocated pointers to the trace, I would also emit relocations
into the trace. For the reasons of port-mortem GPU hang debugging, I
would want the execbuf be complete before the ioctl, rather than post
processing.
 
> What would happen, is that if a batchbuffer made the GPU hang, you would then know all the GL commands (trace ID's from the API trace) that made stuff on that batchbuffer. Then one could go back to the apitrace of the troublesome application  and have a much better starting place to debug.

Yup. As times go on, I hope this becomes a more complete flight-recorder
that we don't have to rely on referencing back to a separate trace to
work out the interesting calls. My goal is that you can give one
instruction (that doesn't require any additional dependencies, so can
just be LD_PRELOAD=i965-fdr.so, or better a script installed in mesa-utils?)
to a bug reporter and that will then capture enough information.
 
> We could also do something evil looking and put another modification on apitrace where it can have a list of call trace ranges where it inserts glFinish after each call. Those glFinish()'s will then force the ioctl of the exact troublesome draw call without needing to tell i965 to flush after each draw call.
> 
> Just to make sure, you want the "apitrace" data (call ID list, maybe function name) in a GEM BO? Which GEM BO should it be in the list so that kernel debug code know which one to use to dump? I would guess if the batchbuffer is the first buffer, then it would be the last buffer, otherwise if the batch buffer is the last one, I guess it would be one just before, but that might screw up reloc-data if any of the relocs in the batchbuffer refer to itself. I can also emit the data to a file and close the file before the ioctl and if the ioctl returns, delete said file (assuming a GPU hang always stops the process, then a hang would leave behind a file). 

My vision is that you would attach all "files" to the execbuf, but then
again I'm focusing on fdr and not debugging of new features. So long
as we are talking about a few megabytes of trace data that isn't too
bad. Then we don't have to fiddle around with extra files to find the
ones corresponding to the hang, as they will be recorded in the error
state. The contents I leave up to you :) (I figure it is a snowball,
once a tracing mechanism exists for capturing GPU hangs, there'll be
lots of suggestions! One is probably just to capture the aub annotations
alongside the batch. Hmm, that might be a good one for me to try just so
I can flesh out the fdr mechanism...)
-Chris