No GTT->VRAM unswapping with amdgpu?

[Alex Deucher]

On Thu, May 11, 2023 at 9:38 PM Linus Lüssing <linus.luessing at c0d3.blue> wrote:
>
> Hi,
>
> Initially noticed this in some games, which seem to be a bit lazy
> with their VRAM allocations + freeing, that performance would drop
> dramatically / become unusable once the VRAM is full and GTT gets
> used. For instance in No Man's Sky after jumping a solar system
> or on DCS World on several maps and in multiplayer. I'm using an
> ATI/AMD Radeon RX 6650 XT, 8GB VRAM with an eGPU enclosure
> (Razer Core X Chroma, Thunderbolt 3), connected via a 40Gbit/s
> USB4 port. Which usually has great performance, until I hit the
> VRAM limit.
>
> I scripted some further tests and benchmarking around
> memmtest_vulkan(*) and these were the results:
>
> https://github.com/T-X/linux-amdgpu-radeon-vram-swapping-test/
>
> I would have expected roughly the same speeds in all tests.
> However tests 5) and 6) yielded significantly lower performance.
>
> Which leads me to the conclusion that anything that gets allocated
> on GTT stays there and is never unswapped from GTT / system memory
> back to VRAM?

It will get swapped back to VRAM if it makes sense for performance.
The driver throttles swapping if there is too much contention to avoid
the overhead of swapping large amounts of memory back and forth
between vram and gtt for every command submission.

Alex

>
> I also read that there was some rework with a new TTM allocator
> in 2020. But dynamic (un)swapping via TTM seems currently unused?
> Is this expected?
>
> Regards, Linus
>
> (*): https://github.com/GpuZelenograd/memtest_vulkan