[rfc] drm/ttm/memcg: simplest initial memcg/ttm integration (v2)

[Christian König]

On 5/26/25 22:13, Dave Airlie wrote:
> On Mon, 26 May 2025 at 18:19, Christian König <christian.koenig at amd.com> wrote:
>>
>> For the HPC/ML use case this feature is completely irrelevant. ROCm, Cuda, OpenCL, OpenMP etc... don't even expose something like this in their higher level APIs as far as I know.
> 
> What do we consider higher level here btw? HIP and CUDA both expose
> something like hipHostMallocWriteCombined, there is also
> hipHostMallocCoherent which may or may not have an effect.

High level in the sense as things which applications actually use.

And yes hip used to have flags for WC memory, but because of allocating system memory through the KFD node never worked as intended we abandoned that approach long time ago.

What could be is that the userspace components (like hip) now use the render node interface with the AMDGPU_GEM_CREATE_CPU_GTT_USWC flag to allocate WC memory. That is one thing I need to double check.

>> The main reason why we still need it for business today is cloud gaming. And for this particular use case you absolutely do want to share the pool between cgroups or otherwise the whole use case breaks.
> 
> I'm still not convinced on this being totally true, which means either
> I'm misunderstanding how cloud gaming works or you are underestimating
> how cgroups work,
> 
> My model for cloud gaming is, you have some sort of orchestrator
> service running that spawns a bunch of games in their own cgroups and
> those games would want to operate as independently as possible.
> 
> Now if the toplevel cgroup or if none the root cgroup exists and the
> game cgroups are all underneath it, then I think this would operate
> more optimally for each game, since
> 
> a) if a game uses uncached memory continuously it will have it's own
> pool of uncached memory that doesn't get used by anyone else, thus
> making that game more consistent.
> b) if and when the game exits, the pool will be returned to the parent
> cgroup to use, this memory should then be reused by other games the
> are started subsequently.

Mhm, my impression was that cgroups are not hierarchically any more these days. So pages would stay allocated to the cgroup until it is fully destroyed and then given back to the core memory management. But I could certainly be wrong on that.

When an application continuously uses write combined memory it would allocate that only once and then re-use it over and over again. The background is that even with the pool we don't make any allocation performance guarantees, the operation is purely best effort.

So the important advantage of the pool is to share the memory between applications which are otherwise not aware of each other. And that is exactly what you have in a cloud gaming use case.

If we would want to make performance guarantees we would need to approach that from a different side, e.g. have per cgroup config options on the pool size, background workers which refills the pool for each cgroup etc... That is certainly possible, I just don't see the use case for that.

One additional side note: A big original motivation for the pool was to avoid the WBINVD CPU instruction, because that is disruptive not only to the current process but to the system as a whole. This is not an issue any more on modern CPUs because we now have the CFLUSH instruction.

So the motivation of having the pool is really not that high any more.

> The only thing I'm not sure is how the parent pool gets used once it's
> built up for new children, need to spend more time reading list_lru
> code.
> 
> The list_lru change might actually be useful for us without cgroups as
> it might be able to hide some of our per-numa stuff.

Yes, having proper NUMA integration would be a really really big argument in favor of having this.

I also explicitly asked that Waiman Long, but it seems like that cgroups doesn't have the NUMA capabilities we actually need here.  

Regards,
Christian.

> 
> Dave.