[RFC] CRIU support for ROCm

Tue May 4 13:00:24 UTC 2021

On Fri, Apr 30, 2021 at 09:57:45PM -0400, Felix Kuehling wrote:
> We have been working on a prototype supporting CRIU (Checkpoint/Restore
> In Userspace) for accelerated compute applications running on AMD GPUs
> using ROCm (Radeon Open Compute Platform). We're happy to finally share
> this work publicly to solicit feedback and advice. The end-goal is to
> get this work included upstream in Linux and CRIU. A short whitepaper
> describing our design and intention can be found on Github:
> https://github.com/RadeonOpenCompute/criu/tree/criu-dev/test/others/ext-kfd/README.md.
>
> We have RFC patch series for the kernel (based on Alex Deucher's
> amd-staging-drm-next branch) and for CRIU including a new plugin and a
> few core CRIU changes. I will send those to the respective mailing lists
> separately in a minute. They can also be found on Github.
>
>     CRIU+plugin: https://github.com/RadeonOpenCompute/criu/commits/criu-dev
>     Kernel (KFD):
>     https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/commits/fxkamd/criu-wip
>
> At this point this is very much a work in progress and not ready for
> upstream inclusion. There are still several missing features, known
> issues, and open questions that we would like to start addressing with
> your feedback.

Since the thread is a bit split I'm dumping the big thoughts here on this
RFC.

We've discussed this in the past, but I'm once more (insert meme here)
asking whether continuing to walk down the amdgpu vs amdkfd split is
really the right choice. It starts to feel a bit much like sunk cost
fallacy ...

- From the big thread we're having right now on dri-devel it's clear that
  3d will also move towards more and more a userspace submit model. But
  due to backwards compat issues it will be a mixed model, and in some
  cases we need to pick at runtime which model we're picking. A hard split
  between the amdgpu and the amdkfd world gets in the way here.

- There's use-cases for doing compute in vulkan (that was a discussion
  from Feb that I kicked again in private, since I think still
  unresolved). So you need a vulkan stack that runs on both amdgpu and
  amdvlk.

- Maybe not yet on amd's radar, but there's a lot of cloud computing. And
  maybe they also want CRIU for migrating their containers around. So that
  means CRIU for amdgpu too, not just amdkf.

- What's much worse, and I don't think anyone in amd has realized this yet
  (at least not in a public thread I've seen). In vulkan you need to be
  able to switch from compute mode to dma-fence mode after
  pipelines/devices have been created already. This is because winsys are
  only initialized in a second step, until that's done you have to
  pessimistically assume that the user does pure compute. What's worse for
  buffer sharing you don't even have a clear signal on this stuff. So
  either

  - you figure out how to migrate all the buffers and state from amdkfd to
    amdgpu at runtime, and duplicate all the features. Which is rather
    pointless.

  - or you duplicate all the compute features to amdgpu so that vk can use
    them, and still reasonably easy migrate to winsys/dma-fence mode,
    which makes amdkfd rather redundant.

  I've discussed this problem extensively with Jason Ekstrand, and it's
  really nasty.

So purely from a technical pov, only looking at the AMD perspective here,
this doesn't make much sense to me. The only reason to keep doubling down
on amdkfd I'm seeing is that you've built your compute rocm stack on top
of it, and because of that the only option is to keep doing that. Which
stops making sense eventually, and we're getting to that point for sure.

The other side is a bit the upstream side, but that's a lot smaller:

- vulkan compute is one of the more reasonable ways to get cross vendor
  compute ecosystem off the ground. At least from what I know from
  background chatter, which you guys probably haven't all heard. amdkfd
  being the single very odd driver here requiring entirely different uapi
  for compute mode is not going to be great.

- CRIU will need new access rights handling (for the save/restore/resume
  stuff you're adding). Generally we standardize access rights checks
  across drivers, and leave everything else to render drivers (command
  submission, memory management, ...). By adding CRIU support to amdkfd
  we pretty much guarantee that we wont be able to standardize CRIU access
  rights across drivers. Which just plains sucks from an
  upstream/cross-vendor ecosystem pov.

And yes we'd still need a per-driver criu plugin in userspace, but the
same is true for amdvlk/radv/anv/ and all the other drivers we have:
Driver is different, access right management is still the same.

And secondly, just because nvidia refuses to collaborate in any
standards around gpu compute doesn't mean that's a good reason for us to
do the same in upstream.

Thirdly, it sounds like this is the first device-driver CRIU support, so I
think we need a solid agreement/standard here to set as an example for
everyone else. There's all the AI accel chips and fpga-for-compute stuff
that I expect will eventually also gain CRIU support.

So I'm asking once more, is this _really_ the right path forward? Both for
amd (at least long term), but also somewhat for upstream.

Cheers, Daniel

> What's working and tested at this point:
>
>   * Checkpoint and restore accelerated machine learning apps: PyTorch
>     running Bert on systems with 1 or 2 GPUs (MI50 or MI100), 100%
>     unmodified user mode stack
>   * Checkpoint on one system, restore on a different system
>   * Checkpoint on one GPU, restore on a different GPU
>
> Major Known issues:
>
>   * The KFD ioctl API is not final: Needs a complete redesign to allow
>     future extension without breaking the ABI
>   * Very slow: Need to implement DMA to dump VRAM contents
>
> Missing or incomplete features:
>
>   * Support for the new KFD SVM API
>   * Check device topology during restore
>   * Checkpoint and restore multiple processes
>   * Support for applications using Mesa for video decode/encode
>   * Testing with more different GPUs and workloads
>
> Big Open questions:
>
>   * What's the preferred way to publish our CRIU plugin? In-tree or
>     out-of-tree?
>   * What's the preferred way to distribute our CRIU plugin? Source?
>     Binary .so? Whole CRIU? Just in-box support?
>   * If our plugin can be upstreamed in the CRIU tree, what would be the
>     right directory?
>
> Best regards,
>   Felix
>
>
> _______________________________________________
> dri-devel mailing list
> dri-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel

--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch