[RFC PATCH 0/8] GPU memory tracepoints

[Kalesh Singh]

On Thu, Oct 21, 2021 at 5:00 AM Daniel Vetter <daniel at ffwll.ch> wrote:
>
> On Wed, Oct 20, 2021 at 08:10:19PM -0700, Gurchetan Singh wrote:
> > This is latest iteration of GPU memory tracepoints [1].
> >
> > In the past, there were questions about the "big picture" of memory
> > accounting [2], especially given related work on dma-buf heaps and DRM
> > cgroups [3].  Also, there was a desire for a non-driver specific solution.
> >
> > The great news is the dma-buf heaps work as recently landed [4].  It uses
> > sys-fs and the plan is to use it in conjunction with the tracepoint
> > solution [5].  We're aiming for the GPU tracepoint to calculate totals
> > per DRM-instance (a proxy for per-process on Android) and per-DRM device.
> >
> > The cgroups work looks terrific too and hopefully we can deduplicate code once
> > that's merged.  Though that's abit of an implementation detail, so long as
> > the "GPU tracepoints" +  "dma-buf heap stats" plan sounds good for Android.
>
> Can we please start out with depulicated code, and integrate this with
> cgroups?

Thanks for the comments Dan,

The cgroups work is currently targeting allocator attribution so it
wouldn’t give insight to shared / imported memory - this is included
as part of the totals in the tracepoint. We will start a separate
discussion with the gpu community on including imported memory into
cgroups design. Who would you recommend to be included? (in case we
don't already know all the interested parties).

The current tracepoint and the cgroups are not conflicting designs but
rather complimentary. These are some of the gaps which the tracepoint
helps to cover:
1. Imported gpu memory accounting
2. The tracepoint can be used to track gpu memory usage over time
(useful to detect memory usage spikes, for example), while cgroups can
be used to view usage as a more granular and static state.
3. For systems where cgroups aren't enabled the tracepoint data can be
a good alternative to identify memory issues.
4. Non-drm devices can make use of the tracepoint for reporting.

It would be great if we can also keep the tracepoint, as we don’t have
another alternative that provides all it offers (cgroups can certainly
be extended to cover some of these),  and it's currently being used by
all Android devices.

Thanks,
Kalesh

>
> The problem with gpu memory account is that everyone wants their own
> thing, they're all slightly differently, and all supported by a different
> subset of drivers. That doesn't make sense to support in upstream at all.
>
> Please huddle together so that there's one set of "track gpu memory"
> calls, and that does cgroups, tracepoints and everything else that an OS
> might want to have.
>
> Also ideally this thing works for both integrated soc gpu (including an
> answer for special memory pools like cma) _and_ discrete gpus using ttm.
> Or at least has an answer to both, because again if we end up with totally
> different tracking for the soc vs the discrete gpu world, we've lost.
> -Daniel
>
> >
> > This series modifies the GPU memory tracepoint API in a non-breaking fashion
> > (patch 1), and adds accounting via the GEM subsystem (patches 2 --> 7). Given
> > the multiple places where memory events happen, there's a bunch trace events
> > scattered in various places.  The hardest part is allocation, where each driver
> > has their own API.  If there's a better way, do say so.
> >
> > The last patch is incomplete; we would like general feedback before proceeding
> > further.
> >
> > [1] https://lore.kernel.org/lkml/20200302235044.59163-1-zzyiwei@google.com/
> > [2] https://lists.freedesktop.org/archives/dri-devel/2021-January/295120.html
> > [3] https://www.spinics.net/lists/cgroups/msg27867.html
> > [4] https://www.spinics.net/lists/linux-doc/msg97788.html
> > [5] https://source.android.com/devices/graphics/implement-dma-buf-gpu-mem
> >
> > Gurchetan Singh (8):
> >   tracing/gpu: modify gpu_mem_total
> >   drm: add new tracepoint fields to drm_device and drm_file
> >   drm: add helper functions for gpu_mem_total and gpu_mem_instance
> >   drm: start using drm_gem_trace_gpu_mem_total
> >   drm: start using drm_gem_trace_gpu_mem_instance
> >   drm: track real and fake imports in drm_prime_member
> >   drm: trace memory import per DRM file
> >   drm: trace memory import per DRM device
> >
> >  drivers/gpu/drm/Kconfig        |  1 +
> >  drivers/gpu/drm/drm_gem.c      | 65 +++++++++++++++++++++++++++++++++-
> >  drivers/gpu/drm/drm_internal.h |  4 +--
> >  drivers/gpu/drm/drm_prime.c    | 22 +++++++++---
> >  include/drm/drm_device.h       | 16 +++++++++
> >  include/drm/drm_file.h         | 16 +++++++++
> >  include/drm/drm_gem.h          |  7 ++++
> >  include/trace/events/gpu_mem.h | 61 +++++++++++++++++++++----------
> >  8 files changed, 166 insertions(+), 26 deletions(-)
> >
> > --
> > 2.25.1
> >
>
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch