No GTT->VRAM unswapping with amdgpu?

[Alex Deucher]

On Mon, May 15, 2023 at 10:42 AM Alex Deucher <alexdeucher at gmail.com> wrote:
>
> On Thu, May 11, 2023 at 9:38 PM Linus Lüssing <linus.luessing at c0d3.blue> wrote:
> >
> > Hi,
> >
> > Initially noticed this in some games, which seem to be a bit lazy
> > with their VRAM allocations + freeing, that performance would drop
> > dramatically / become unusable once the VRAM is full and GTT gets
> > used. For instance in No Man's Sky after jumping a solar system
> > or on DCS World on several maps and in multiplayer. I'm using an
> > ATI/AMD Radeon RX 6650 XT, 8GB VRAM with an eGPU enclosure
> > (Razer Core X Chroma, Thunderbolt 3), connected via a 40Gbit/s
> > USB4 port. Which usually has great performance, until I hit the
> > VRAM limit.
> >
> > I scripted some further tests and benchmarking around
> > memmtest_vulkan(*) and these were the results:
> >
> > https://github.com/T-X/linux-amdgpu-radeon-vram-swapping-test/
> >
> > I would have expected roughly the same speeds in all tests.
> > However tests 5) and 6) yielded significantly lower performance.
> >
> > Which leads me to the conclusion that anything that gets allocated
> > on GTT stays there and is never unswapped from GTT / system memory
> > back to VRAM?
>
> It will get swapped back to VRAM if it makes sense for performance.
> The driver throttles swapping if there is too much contention to avoid
> the overhead of swapping large amounts of memory back and forth
> between vram and gtt for every command submission.

RADV also dynamically prefers GTT if you are using an eGPU and memory
is contended:
https://cgit.freedesktop.org/mesa/mesa/commit/?id=95d06343c693aa12b4cda5cda31d81fae138b0ec

Alex

>
> Alex
>
> >
> > I also read that there was some rework with a new TTM allocator
> > in 2020. But dynamic (un)swapping via TTM seems currently unused?
> > Is this expected?
> >
> > Regards, Linus
> >
> > (*): https://github.com/GpuZelenograd/memtest_vulkan