[RFC] CRIU support for ROCm

[Felix Kuehling]

Am 2021-05-04 um 9:00 a.m. schrieb Daniel Vetter:
> On Fri, Apr 30, 2021 at 09:57:45PM -0400, Felix Kuehling wrote:
>> We have been working on a prototype supporting CRIU (Checkpoint/Restore
>> In Userspace) for accelerated compute applications running on AMD GPUs
>> using ROCm (Radeon Open Compute Platform). We're happy to finally share
>> this work publicly to solicit feedback and advice. The end-goal is to
>> get this work included upstream in Linux and CRIU. A short whitepaper
>> describing our design and intention can be found on Github:
>> https://github.com/RadeonOpenCompute/criu/tree/criu-dev/test/others/ext-kfd/README.md
>>
>> We have RFC patch series for the kernel (based on Alex Deucher's
>> amd-staging-drm-next branch) and for CRIU including a new plugin and a
>> few core CRIU changes. I will send those to the respective mailing lists
>> separately in a minute. They can also be found on Github.
>>
>>     CRIU+plugin: https://github.com/RadeonOpenCompute/criu/commits/criu-dev
>>     Kernel (KFD):
>>     https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/commits/fxkamd/criu-wip
>>
>> At this point this is very much a work in progress and not ready for
>> upstream inclusion. There are still several missing features, known
>> issues, and open questions that we would like to start addressing with
>> your feedback.
> Since the thread is a bit split I'm dumping the big thoughts here on this
> RFC.
>
> We've discussed this in the past, but I'm once more (insert meme here)
> asking whether continuing to walk down the amdgpu vs amdkfd split is
> really the right choice. It starts to feel a bit much like sunk cost
> fallacy ...

Hi Daniel,

Thanks for the feedback. I have some comments to your specific points
below. This is my own opinion at this point and may not reflect AMDs
position. I'm starting some internal discussions about unifying the KFD
and graphics APIs in the long run. But IMO this is going to take years
and won't be supported on our current compute GPUs, including Aldebaran
which isn't even released yet.


>
> - From the big thread we're having right now on dri-devel it's clear that
>   3d will also move towards more and more a userspace submit model.

I'll need to start following dri-devel more closely and take a more
active role in those discussions. If there is an opportunity for a
unified memory management and command submission model for graphics and
compute on future hardware, I want to be sure that our compute
requirements are understood early on.


>  But
>   due to backwards compat issues it will be a mixed model, and in some
>   cases we need to pick at runtime which model we're picking. A hard split
>   between the amdgpu and the amdkfd world gets in the way here.

Backwards compatibility will force us to maintain KFD at least for GFXv9
and older AMD GPUs. As I understand it, the new user mode command
submission model will only be viable on GFXv10 or even newer GPUs. GFXv9
is our architecture for current compute GPUs (including Aldebaran which
is still in development), so this is going to be important for us for
years to come.


>
> - There's use-cases for doing compute in vulkan (that was a discussion
>   from Feb that I kicked again in private, since I think still
>   unresolved). So you need a vulkan stack that runs on both amdgpu and
>   amdvlk.

By amdvlk, do you mean AMD's "pro" driver that supports our non-RADV
Vulcan driver?


>
> - Maybe not yet on amd's radar, but there's a lot of cloud computing. And
>   maybe they also want CRIU for migrating their containers around. So that
>   means CRIU for amdgpu too, not just amdkf.

Our strategy for compute support (OpenCL, HIP, OpenMP and mayby future
C++ standards) is to use ROCm on all current and future GPUs starting
with GFXv9. As long as ROCm is based on KFD, that means our compute
stack runs on KFD. We don't see KFD as something mutually exclusive with
graphics. They are different ioctl APIs, but they're sharing the
hardware and memory and are meant to work together.

We are already planning to add CRIU support to our render node API
because video acceleration on our compute stack still relies on Mesa and
the render node CS API.

I admit, that we are not currently considering Vulcan as part of our
compute stack. I don't think it makes sense to build Vulcan on top of
ROCm because we have no plans to add graphics support to ROCm. What
commonality there is between Vulcan and other compute APIs will need to
be addressed at a lower level. If graphics is moving to user mode
command submission and a shared virtual memory model, I see an
opportunity to move ROCm to a common ioctl API on those future hardware
generations.


>
> - What's much worse, and I don't think anyone in amd has realized this yet
>   (at least not in a public thread I've seen). In vulkan you need to be
>   able to switch from compute mode to dma-fence mode after
>   pipelines/devices have been created already. This is because winsys are
>   only initialized in a second step, until that's done you have to
>   pessimistically assume that the user does pure compute. What's worse for
>   buffer sharing you don't even have a clear signal on this stuff. So
>   either
>
>   - you figure out how to migrate all the buffers and state from amdkfd to
>     amdgpu at runtime, and duplicate all the features. Which is rather
>     pointless.
>
>   - or you duplicate all the compute features to amdgpu so that vk can use
>     them, and still reasonably easy migrate to winsys/dma-fence mode,
>     which makes amdkfd rather redundant.

The basic "compute features" that ROCm depends on are a
shared-virtual-memory model and user mode command submission. If those
become part of the graphics ioctl API, I see no problem with
implementing proper Vulcan compute on top of it.

For more consistent support for tools such as debugger, profilers, CRIU,
etc. it would make sense to have a common ioctl API for ROCm and Vulcan,
which would effectively deprecate the KFD API. And I'm definitely open
to working on that for future HW generations that can support it.

We'll probably have to pick a target GPU where we make such a switch,
and use the time until then to get things ready, prototyping in current
GFXv10 or newer GPUs. This feels to me a bit like the transition from
radeon to amdgpu, where GFXv7 was supported by both for some time.

 

>
>   I've discussed this problem extensively with Jason Ekstrand, and it's
>   really nasty.
>
> So purely from a technical pov, only looking at the AMD perspective here,
> this doesn't make much sense to me. The only reason to keep doubling down
> on amdkfd I'm seeing is that you've built your compute rocm stack on top
> of it, and because of that the only option is to keep doing that. Which
> stops making sense eventually, and we're getting to that point for sure.

As long as our compute GPUs are based on the GFXv9 architecture, I think
we don't really have an alternative than to double down on KFD. We can't
just stop work on KFD and tell our customers that they'll not get any
new features for the next 2 or 3 years that it takes to build a common
upstream memory management and scheduling API and the HW that can
support it.


>
> The other side is a bit the upstream side, but that's a lot smaller:
>
> - vulkan compute is one of the more reasonable ways to get cross vendor
>   compute ecosystem off the ground. At least from what I know from
>   background chatter, which you guys probably haven't all heard. amdkfd
>   being the single very odd driver here requiring entirely different uapi
>   for compute mode is not going to be great.

Like I said, KFD is not exclusive with anything else.

Vulcan may be a good option for desktops. For HPC uses or general
portable non-graphics compute code, I think something like OpenMP or
future C++ standards would be a better way forward. There is no
technical reason why both cannot coexist in the same application.


>
> - CRIU will need new access rights handling (for the save/restore/resume
>   stuff you're adding). Generally we standardize access rights checks
>   across drivers, and leave everything else to render drivers (command
>   submission, memory management, ...). By adding CRIU support to amdkfd
>   we pretty much guarantee that we wont be able to standardize CRIU access
>   rights across drivers. Which just plains sucks from an
>   upstream/cross-vendor ecosystem pov.

By access rights, do you mean requiring root for some ioctls, or being
ptrace-attached for others? These are driven by how CRIU works and
interacts with its target processes. I think they will apply equally to
any driver implementing CRIU support. I don't see how graphics drivers
specifically drive standardization of CRIU access rights.


>
> And yes we'd still need a per-driver criu plugin in userspace, but the
> same is true for amdvlk/radv/anv/ and all the other drivers we have:
> Driver is different, access right management is still the same.
>
> And secondly, just because nvidia refuses to collaborate in any
> standards around gpu compute doesn't mean that's a good reason for us to
> do the same in upstream.
>
> Thirdly, it sounds like this is the first device-driver CRIU support, so I
> think we need a solid agreement/standard here to set as an example for
> everyone else. There's all the AI accel chips and fpga-for-compute stuff
> that I expect will eventually also gain CRIU support.

Sure. My experience with CRIU so far is, that CRIU requires root (even
without our plugin). Regardless of that, CRIU attaches to the target
process with ptrace. The kernel's security policy for allowing ptrace
access is influenced by many factors. We chose not to duplicate this in
the driver. Therefore we check whether the caller of our CRIU dump ioctl
is ptrace attached. If it is, it means it has passed all the kernel's
security checks. We deliberately do not want to implement our own
security policy in the driver.

There would be ways to remove the root requirement from our CRIU
restore-ioctl, if that becomes a requirement for CRIU. It would mean
some sanity checking of the HW state that's getting restored.


>
> So I'm asking once more, is this _really_ the right path forward? Both for
> amd (at least long term), but also somewhat for upstream.

I think long-term it makes sense to plan for a unified API. For current
hardware I don't see a way to get there for multiple reasons:

 1. Maintaining backwards compatibility with existing user mode
 2. Lacking HW capabilities for graphics user mode command submission
    and HW scheduling

Therefore KFD will be important for AMD's compute GPU strategy and our
customers for years to come. Not being able to continue developing it
upstream would be a major impediment for AMD. Again, this is my personal
opinion.

Regards,
  Felix


>
> Cheers, Daniel
>
>
>> What's working and tested at this point:
>>
>>   * Checkpoint and restore accelerated machine learning apps: PyTorch
>>     running Bert on systems with 1 or 2 GPUs (MI50 or MI100), 100%
>>     unmodified user mode stack
>>   * Checkpoint on one system, restore on a different system
>>   * Checkpoint on one GPU, restore on a different GPU
>>
>> Major Known issues:
>>
>>   * The KFD ioctl API is not final: Needs a complete redesign to allow
>>     future extension without breaking the ABI
>>   * Very slow: Need to implement DMA to dump VRAM contents
>>
>> Missing or incomplete features:
>>
>>   * Support for the new KFD SVM API
>>   * Check device topology during restore
>>   * Checkpoint and restore multiple processes
>>   * Support for applications using Mesa for video decode/encode
>>   * Testing with more different GPUs and workloads
>>
>> Big Open questions:
>>
>>   * What's the preferred way to publish our CRIU plugin? In-tree or
>>     out-of-tree?
>>   * What's the preferred way to distribute our CRIU plugin? Source?
>>     Binary .so? Whole CRIU? Just in-box support?
>>   * If our plugin can be upstreamed in the CRIU tree, what would be the
>>     right directory?
>>
>> Best regards,
>>   Felix
>>
>>
>> _______________________________________________
>> dri-devel mailing list
>> dri-devel at lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/dri-devel
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch/