[Mesa-dev] [PATCH 00/18] Batchbuffer Logger for Intel GPU (v2)

kevin.rogovin at intel.com kevin.rogovin at intel.com
Mon Nov 13 13:17:48 UTC 2017

From: Kevin Rogovin <kevin.rogovin at intel.com>

This patch series defines and implements a BatchbufferLogger
for Intel GEN. The main purpose of the BatchbufferLogger is
to strongly correlate API calls to data added to a batchbuffer.
In addition to this function, the BatchbufferLogger also tracks
GPU state (respecting HW context as well). The logger intercepts
drmIoctl recording the information needed to decode a bachbuffer
(such as GEM BO creation/deletion, HW context create/delete,
and most importantly execbuffer2). When the execbuffer2 returns
from the kernel, the BatchbufferLogger will log information
in its log of what was added when and in addition log the
GPU state (at the point in the batchbuffer) of 3DPRIMITIVE and
GPGPU_WALKER commands.

It is an application's requirment to tell the BatchbufferLogger
just before and just after an API call. Because of the need
to intercept drmIoctl, having an application link against
BatchbufferLogger is not robust. Instead, an application is
to use dlsym to fetch the correction a function pointer that
returns the BatchbufferLogger's application interface. The
interface of the BatchbufferLogger is defined in patch 0002.
A script is also provided to use the BatchbufferLogger in
an easier way than needing to set environmental variables.

On the subject of application integration, I have a -very-
small patch series that enabled BatchbufferLogger for
apitrace. I can share these patches if anyone asks, but I
cannot submit them to apitrace until atleast the BatchbufferLogger
is in Mesa with a stable application interface.

The log emitted by the BatchbufferLogger is a sequence of blocks
with possibility of blocks being elements of blocks. The top level
blocks are the API call markers created from the calls into the
BatchbufferLogger from the application. An application starts
and ends a logging session. A session provides callbacks to
write the block data; with the interface of a callback a log
does not need to just write to a file, it could also write
over a network to remote application for the purpose of remote
debugging. A specific interface is also provided to write to a
file and additional tools are included to convert the log to JSON,
XML and text. The simple file format should allow others to be able
to take the data and use it however they see fit. The JSON output
alone can be quite illuminating to use when debugging/enhancing the
i965 driver for a single frame (or even single draw call) of a troublesome

It is worth noting that i965 already has a batchbuffer decoder
dumper (also making use src/intel/tools/gen_decoder). However,
BatchbufferLogger has the following features which are not in the
current dumper and are likely quite tricky to implement into it.

 1. The biggest feature is that this BatchbufferLogger contents are
    annotated by the GL/GLES API calls and driver text as well.
 2. This BatchbufferLogger provides an interface to application space
    to allow for applications to start/end logging sessions as
    they see fit. In addition, an application can have multilpe
    independent sessions active.
 3. This BatchbufferLogger repeats the state of the GPU and HW
    context on 3DPRIMITIVE and GPGPU_WALKER commands. By doing
    so one can see the precise state the GPU is in at that command.
    Coupled with (1), one can see what precise GPU state one
    has when an application issues draw commands. Moreover, coupled
    with the (previously) posted patches for apitrace, one
    can compare differences at specific trace points within a
    trace from apitrace to help diagnose regressions.

In addition, as suggested by Chris Wilson, I have also implemented an
i965 blackbox recorded at https://github.com/krogueintel/i965-blackbox
which will perform the necessary GL/GLES API interception to
produce annotated batchbuffer recordings to assist in GPU hang
debugging. The BatchbufferLogger (again at Chris Wilson's suggestion)
has the ability to walk batchbuffers BEFORE kernel submit and to
also add an EXEC_OBJECT_CAPTURE execobject2 whose contents are
the ID of the execbuffer2 ioctl submitted. The aim is to greatly
assist in GPU hang debugging for troublesome applications.

For those interested, this BatchbufferLogger is available on
github at https://github.com/krogueintel/asem/tree/batchbuffer-logger.

The patch series is organized into the following blocks:

0001-0003: Define the BatchbufferLogger interfaces
0004-0004: Minor fix to i965 driver
0005-0005: Hooking of BatchbufferLogger into i965
0006-0011: Fixes and enhancements to intel/compiler,
           intel/tools and intel/common.
0012-0014: Implementation of BatchBufferLogger
0015-0017: Tools to decode log to JSON, XML and text
0018-0018: Command line tool for disassembling shader

  bug fix (spotted by Chris Wilson) for when i965 moves batchbuffer content
  add application session interface for more customizable logging
  add driver interface to add annotation to batchbuffer logs
  add option to log batchbuffer contents before execbuffer2 ioctl
  add option to emit EXEC_OBJECT_CAPTURE exec_object2
  simpify from each shader type to decode or not to same rule for all stages
  numerous tweaks and improvements in BatchbufferLogger implementation

Kevin Rogovin (18):
  intel/tools: define BatchBufferLogger driver interface
  intel/tools: define BatchbufferLogger application interface
  intel/tools: BatchBufferLogger define output file format of tool
  i965: assign BindingTableEntryCount of INTERFACE_DESCRIPTOR_DATA
  i965: Enable BatchbufferLogger in i965 driver
  intel/common/gen_decoder: make useable from C++ source
  intel/compiler: fix for memmove argument on annotating error
  intel/compiler:add function to give option to print offsets into
  intel/tools/disasm: gen_disasm_disassemble to take const void* instead
    of void*
  intel/tools/disasm: add gen_disasm_assembly_length function
  intel/tools/disasm: make sure that entire range is disassembled
  intel/tools/BatchbufferLogger: first implementation
  intel/tools/BatchbufferLogger: install i965_batchbuffer non-driver
    interface headers
  intel/tools/BatchbufferLogger : add shell script for batchbuffer
  intel/tools/BatchbufferLogger (txt-output): example txt dumper
  intel/tools/BatchbufferLogger (output-xml): add outputter to XML
  intel/tools/BatchbufferLogger (output-json): add json outputter
  intel/tools: add command line GEN shader disassembler tool

 src/intel/Makefile.tools.am                        |   71 +
 src/intel/common/gen_decoder.h                     |    7 +
 src/intel/compiler/brw_eu.c                        |   11 +-
 src/intel/compiler/brw_eu.h                        |    3 +
 src/intel/compiler/intel_asm_annotation.c          |    5 +-
 src/intel/tools/.gitignore                         |    5 +
 src/intel/tools/disasm.c                           |   26 +-
 src/intel/tools/gen_disasm.h                       |    4 +-
 src/intel/tools/gen_shader_disassembler.c          |  221 +
 src/intel/tools/i965_batchbuffer_dump_show.c       |  129 +
 .../tools/i965_batchbuffer_dump_show_json.cpp      |  253 +
 src/intel/tools/i965_batchbuffer_dump_show_xml.cpp |  217 +
 src/intel/tools/i965_batchbuffer_logger.cpp        | 6118 ++++++++++++++++++++
 src/intel/tools/i965_batchbuffer_logger.h          |  186 +
 src/intel/tools/i965_batchbuffer_logger_app.h      |  157 +
 .../tools/i965_batchbuffer_logger_instructions.h   |  131 +
 src/intel/tools/i965_batchbuffer_logger_output.h   |   66 +
 src/intel/tools/i965_batchbuffer_logger_sh.in      |   91 +
 src/mesa/drivers/dri/i965/brw_bufmgr.c             |   22 +-
 src/mesa/drivers/dri/i965/brw_bufmgr.h             |    8 +-
 src/mesa/drivers/dri/i965/brw_context.c            |   34 +
 src/mesa/drivers/dri/i965/brw_context.h            |   12 +
 src/mesa/drivers/dri/i965/genX_state_upload.c      |    1 +
 src/mesa/drivers/dri/i965/intel_batchbuffer.c      |   29 +-
 src/mesa/drivers/dri/i965/intel_screen.c           |   46 +-
 src/mesa/drivers/dri/i965/intel_screen.h           |    3 +
 26 files changed, 7837 insertions(+), 19 deletions(-)
 create mode 100644 src/intel/tools/gen_shader_disassembler.c
 create mode 100644 src/intel/tools/i965_batchbuffer_dump_show.c
 create mode 100644 src/intel/tools/i965_batchbuffer_dump_show_json.cpp
 create mode 100644 src/intel/tools/i965_batchbuffer_dump_show_xml.cpp
 create mode 100644 src/intel/tools/i965_batchbuffer_logger.cpp
 create mode 100644 src/intel/tools/i965_batchbuffer_logger.h
 create mode 100644 src/intel/tools/i965_batchbuffer_logger_app.h
 create mode 100644 src/intel/tools/i965_batchbuffer_logger_instructions.h
 create mode 100644 src/intel/tools/i965_batchbuffer_logger_output.h
 create mode 100644 src/intel/tools/i965_batchbuffer_logger_sh.in


More information about the mesa-dev mailing list