[PATCH 0/7] Per client engine busyness

[Daniel Vetter]

On Thu, May 13, 2021 at 11:48:08AM -0400, Alex Deucher wrote:
> On Thu, May 13, 2021 at 7:00 AM Tvrtko Ursulin
> <tvrtko.ursulin at linux.intel.com> wrote:
> >
> > From: Tvrtko Ursulin <tvrtko.ursulin at intel.com>
> >
> > Resurrect of the previosuly merged per client engine busyness patches. In a
> > nutshell it enables intel_gpu_top to be more top(1) like useful and show not
> > only physical GPU engine usage but per process view as well.
> >
> > Example screen capture:
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > intel-gpu-top -  906/ 955 MHz;    0% RC6;  5.30 Watts;      933 irqs/s
> >
> >       IMC reads:     4414 MiB/s
> >      IMC writes:     3805 MiB/s
> >
> >           ENGINE      BUSY                                      MI_SEMA MI_WAIT
> >      Render/3D/0   93.46% |████████████████████████████████▋  |      0%      0%
> >        Blitter/0    0.00% |                                   |      0%      0%
> >          Video/0    0.00% |                                   |      0%      0%
> >   VideoEnhance/0    0.00% |                                   |      0%      0%
> >
> >   PID            NAME  Render/3D      Blitter        Video      VideoEnhance
> >  2733       neverball |██████▌     ||            ||            ||            |
> >  2047            Xorg |███▊        ||            ||            ||            |
> >  2737        glxgears |█▍          ||            ||            ||            |
> >  2128           xfwm4 |            ||            ||            ||            |
> >  2047            Xorg |            ||            ||            ||            |
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > Internally we track time spent on engines for each struct intel_context, both
> > for current and past contexts belonging to each open DRM file.
> >
> > This can serve as a building block for several features from the wanted list:
> > smarter scheduler decisions, getrusage(2)-like per-GEM-context functionality
> > wanted by some customers, setrlimit(2) like controls, cgroups controller,
> > dynamic SSEU tuning, ...
> >
> > To enable userspace access to the tracked data, we expose time spent on GPU per
> > client and per engine class in sysfs with a hierarchy like the below:
> >
> >         # cd /sys/class/drm/card0/clients/
> >         # tree
> >         .
> >         ├── 7
> >         │   ├── busy
> >         │   │   ├── 0
> >         │   │   ├── 1
> >         │   │   ├── 2
> >         │   │   └── 3
> >         │   ├── name
> >         │   └── pid
> >         ├── 8
> >         │   ├── busy
> >         │   │   ├── 0
> >         │   │   ├── 1
> >         │   │   ├── 2
> >         │   │   └── 3
> >         │   ├── name
> >         │   └── pid
> >         └── 9
> >             ├── busy
> >             │   ├── 0
> >             │   ├── 1
> >             │   ├── 2
> >             │   └── 3
> >             ├── name
> >             └── pid
> >
> > Files in 'busy' directories are numbered using the engine class ABI values and
> > they contain accumulated nanoseconds each client spent on engines of a
> > respective class.
> 
> We did something similar in amdgpu using the gpu scheduler.  We then
> expose the data via fdinfo.  See
> https://cgit.freedesktop.org/drm/drm-misc/commit/?id=1774baa64f9395fa884ea9ed494bcb043f3b83f5
> https://cgit.freedesktop.org/drm/drm-misc/commit/?id=874442541133f78c78b6880b8cc495bab5c61704

Yeah the reason I've dropped these patches was because they looked like
prime material for at least a bit of standardization across drivers.

Also fdinfo sounds like very good interface for these, I didn't even know
that's doable. Might also be interesting to even standardize the fdinfo
stuff across drivers.

Also since drm/i915 will adopt drm/scheduler, we could build that on top
of that code too. So no restrictions there from i915 side.

Anyway discussion kicked off, I'll let yout figure out what we'll do here.
-Daniel

> 
> Alex
> 
> 
> >
> > Tvrtko Ursulin (7):
> >   drm/i915: Expose list of clients in sysfs
> >   drm/i915: Update client name on context create
> >   drm/i915: Make GEM contexts track DRM clients
> >   drm/i915: Track runtime spent in closed and unreachable GEM contexts
> >   drm/i915: Track all user contexts per client
> >   drm/i915: Track context current active time
> >   drm/i915: Expose per-engine client busyness
> >
> >  drivers/gpu/drm/i915/Makefile                 |   5 +-
> >  drivers/gpu/drm/i915/gem/i915_gem_context.c   |  61 ++-
> >  .../gpu/drm/i915/gem/i915_gem_context_types.h |  16 +-
> >  drivers/gpu/drm/i915/gt/intel_context.c       |  27 +-
> >  drivers/gpu/drm/i915/gt/intel_context.h       |  15 +-
> >  drivers/gpu/drm/i915/gt/intel_context_types.h |  24 +-
> >  .../drm/i915/gt/intel_execlists_submission.c  |  23 +-
> >  .../gpu/drm/i915/gt/intel_gt_clock_utils.c    |   4 +
> >  drivers/gpu/drm/i915/gt/intel_lrc.c           |  27 +-
> >  drivers/gpu/drm/i915/gt/intel_lrc.h           |  24 ++
> >  drivers/gpu/drm/i915/gt/selftest_lrc.c        |  10 +-
> >  drivers/gpu/drm/i915/i915_drm_client.c        | 365 ++++++++++++++++++
> >  drivers/gpu/drm/i915/i915_drm_client.h        | 123 ++++++
> >  drivers/gpu/drm/i915/i915_drv.c               |   6 +
> >  drivers/gpu/drm/i915/i915_drv.h               |   5 +
> >  drivers/gpu/drm/i915/i915_gem.c               |  21 +-
> >  drivers/gpu/drm/i915/i915_gpu_error.c         |  31 +-
> >  drivers/gpu/drm/i915/i915_gpu_error.h         |   2 +-
> >  drivers/gpu/drm/i915/i915_sysfs.c             |   8 +
> >  19 files changed, 716 insertions(+), 81 deletions(-)
> >  create mode 100644 drivers/gpu/drm/i915/i915_drm_client.c
> >  create mode 100644 drivers/gpu/drm/i915/i915_drm_client.h
> >
> > --
> > 2.30.2
> >

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch