[RFC PATCH 00/13] SVM (share virtual memory) with HMM in nouveau

Sat Mar 10 03:21:28 UTC 2018

From: Jérôme Glisse <jglisse at redhat.com>

(mm is cced just to allow exposure of device driver work without ccing
a long list of peoples. I do not think there is anything usefull to
discuss from mm point of view but i might be wrong, so just for the
curious :)).

git://people.freedesktop.org/~glisse/linux branch: nouveau-hmm-v00
https://cgit.freedesktop.org/~glisse/linux/log/?h=nouveau-hmm-v00

This patchset adds SVM (Share Virtual Memory) using HMM (Heterogeneous
Memory Management) to the nouveau driver. SVM means that GPU threads
spawn by GPU driver for a specific user process can access any valid
CPU address in that process. A valid pointer is a pointer inside an
area coming from mmap of private, share or regular file. Pointer to
a mmap of a device file or special file are not supported.

This is an RFC for few reasons technical reasons listed below and also
because we are still working on a proper open source userspace (namely
a OpenCL 2.0 for nouveau inside mesa). Open source userspace being a
requirement for the DRM subsystem. I pushed in [1] a simple standalone
program that can be use to test SVM through HMM with nouveau. I expect
we will have a somewhat working userspace in the coming weeks, work
being well underway and some patches have already been posted on mesa
mailing list.

They are two aspect that need to sorted before this can be considered
ready. First we want to decide how to update GPU page table from HMM.
In this patchset i added new methods to vmm to allow GPU page table to
be updated without nvkm_memory or nvkm_vma object (see patch 7 and 8
special mapping method for HMM). It just take an array of pages and
flags. It allow for both system and device private memory to be
interleaved.

The second aspect is how to create a HMM enabled channel. Channel is
a term use for NVidia GPU command queue, each process using nouveau
have at least one channel, it can have multiple channels. They are
not created by process directly but rather by device driver backend
of common library like OpenGL, OpenCL or Vulkan.

They are work underway to revamp nouveau channel creation with a new
userspace API. So we might want to delay upstreaming until this lands.
We can stil discuss one aspect specific to HMM here namely the issue
around GEM objects used for some specific part of the GPU. Some engine
inside the GPU (engine are a GPU block like the display block which
is responsible of scaning memory to send out a picture through some
connector for instance HDMI or DisplayPort) can only access memory
with virtual address below (1 << 40). To accomodate those we need to
create a "hole" inside the process address space. This patchset have
a hack for that (patch 13 HACK FOR HMM AREA), it reserves a range of
device file offset so that process can mmap this range with PROT_NONE
to create a hole (process must make sure the hole is below 1 << 40).
I feel un-easy of doing it this way but maybe it is ok with other
folks.

Note that this patchset do not show usage of device private memory as
it depends on other architectural changes to nouveau. However it is
very easy to add it with some gross hack so if people would like to
see it i can also post an RFC for that. As a preview it only adds two
new ioctl which allow userspace to ask for migration of a range of
virtual address, expectation is that the userspace library will know
better where to place thing and kernel will try to sastify this (with
no guaranty, it is a best effort).

As usual comments and questions are welcome.

Cheers,
Jérôme Glisse

[1] https://cgit.freedesktop.org/~glisse/moche

Ben Skeggs (4):
  drm/nouveau/core: define engine for handling replayable faults
  drm/nouveau/mmu/gp100: allow gcc/tex to generate replayable faults
  drm/nouveau/mc/gp100-: handle replayable fault interrupt
  drm/nouveau/fault/gp100: initial implementation of MaxwellFaultBufferA

Jérôme Glisse (9):
  drm/nouveau/vmm: enable page table iterator over non populated range
  drm/nouveau/core/memory: add some useful accessor macros
  drm/nouveau: special mapping method for HMM
  drm/nouveau: special mapping method for HMM (user interface)
  drm/nouveau: add SVM through HMM support to nouveau client
  drm/nouveau: add HMM area creation
  drm/nouveau: add HMM area creation user interface
  drm/nouveau: HMM area creation helpers for nouveau client
  drm/nouveau: HACK FOR HMM AREA

 drivers/gpu/drm/nouveau/Kbuild                     |   3 +
 drivers/gpu/drm/nouveau/include/nvif/class.h       |   2 +
 drivers/gpu/drm/nouveau/include/nvif/clb069.h      |   8 +
 drivers/gpu/drm/nouveau/include/nvif/if000c.h      |  26 ++
 drivers/gpu/drm/nouveau/include/nvif/vmm.h         |   4 +
 drivers/gpu/drm/nouveau/include/nvkm/core/device.h |   3 +
 drivers/gpu/drm/nouveau/include/nvkm/core/memory.h |   8 +
 .../gpu/drm/nouveau/include/nvkm/engine/fault.h    |   5 +
 drivers/gpu/drm/nouveau/include/nvkm/subdev/mmu.h  |  10 +
 drivers/gpu/drm/nouveau/nouveau_drm.c              |   5 +
 drivers/gpu/drm/nouveau/nouveau_drv.h              |   3 +
 drivers/gpu/drm/nouveau/nouveau_hmm.c              | 367 +++++++++++++++++++++
 drivers/gpu/drm/nouveau/nouveau_hmm.h              |  64 ++++
 drivers/gpu/drm/nouveau/nouveau_ttm.c              |   9 +-
 drivers/gpu/drm/nouveau/nouveau_vmm.c              |  83 +++++
 drivers/gpu/drm/nouveau/nouveau_vmm.h              |  12 +
 drivers/gpu/drm/nouveau/nvif/vmm.c                 |  80 +++++
 drivers/gpu/drm/nouveau/nvkm/core/subdev.c         |   1 +
 drivers/gpu/drm/nouveau/nvkm/engine/Kbuild         |   1 +
 drivers/gpu/drm/nouveau/nvkm/engine/device/base.c  |   8 +
 drivers/gpu/drm/nouveau/nvkm/engine/device/priv.h  |   1 +
 drivers/gpu/drm/nouveau/nvkm/engine/device/user.c  |   1 +
 drivers/gpu/drm/nouveau/nvkm/engine/fault/Kbuild   |   4 +
 drivers/gpu/drm/nouveau/nvkm/engine/fault/base.c   | 116 +++++++
 drivers/gpu/drm/nouveau/nvkm/engine/fault/gp100.c  |  61 ++++
 drivers/gpu/drm/nouveau/nvkm/engine/fault/priv.h   |  29 ++
 drivers/gpu/drm/nouveau/nvkm/engine/fault/user.c   | 136 ++++++++
 drivers/gpu/drm/nouveau/nvkm/engine/fault/user.h   |   7 +
 drivers/gpu/drm/nouveau/nvkm/subdev/mc/gp100.c     |  20 +-
 drivers/gpu/drm/nouveau/nvkm/subdev/mc/gp10b.c     |   2 +-
 drivers/gpu/drm/nouveau/nvkm/subdev/mc/priv.h      |   2 +
 drivers/gpu/drm/nouveau/nvkm/subdev/mmu/uvmm.c     |  88 ++++-
 drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.c      | 241 ++++++++++++--
 drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmm.h      |   8 +
 drivers/gpu/drm/nouveau/nvkm/subdev/mmu/vmmgp100.c |  78 ++++-
 35 files changed, 1463 insertions(+), 33 deletions(-)
 create mode 100644 drivers/gpu/drm/nouveau/include/nvif/clb069.h
 create mode 100644 drivers/gpu/drm/nouveau/include/nvkm/engine/fault.h
 create mode 100644 drivers/gpu/drm/nouveau/nouveau_hmm.c
 create mode 100644 drivers/gpu/drm/nouveau/nouveau_hmm.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/engine/fault/Kbuild
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/engine/fault/base.c
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/engine/fault/gp100.c
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/engine/fault/priv.h
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/engine/fault/user.c
 create mode 100644 drivers/gpu/drm/nouveau/nvkm/engine/fault/user.h

-- 
2.14.3