Plan: BO move throttling for visible VRAM evictions

[Marek Olšák]

On Fri, Mar 24, 2017 at 5:45 PM, Christian König
<deathsimple at vodafone.de> wrote:
> Am 24.03.2017 um 17:33 schrieb Marek Olšák:
>>
>> Hi,
>>
>> I'm sharing this idea here, because it's something that has been
>> decreasing our performance a lot recently, for example:
>>
>> http://openbenchmarking.org/prospect/1703011-RI-RADEONDIR06/7b7668cfc109d1c3dc27e871c8aea71ca13f23fa
>>
>> I think the problem there is that Mesa git started uploading
>> descriptors and uniforms to VRAM, which helps when TC L2 has a low
>> hit/miss ratio, but the performance can randomly drop by an order of
>> magnitude. I've heard rumours that kernel 4.11 has an improved
>> allocator that should perform better, but the situation is still far
>> from ideal.
>>
>> AMD CPUs and APUs will hopefully suffer less, because we can resize
>> the visible VRAM with the help of our CPU hw specs, but Intel CPUs
>> will remain limited to 256 MB. The following plan describes how to do
>> throttling for visible VRAM evictions.
>>
>>
>> 1) Theory
>>
>> Initially, the driver doesn't care about where buffers are in VRAM,
>> because VRAM buffers are only moved to visible VRAM on CPU page faults
>> (when the CPU touches the buffer memory but the memory is in the
>> invisible part of VRAM). When it happens,
>> amdgpu_bo_fault_reserve_notify is called, which moves the buffer to
>> visible VRAM, and the app continues. amdgpu_bo_fault_reserve_notify
>> also marks the buffer as contiguous, which makes memory fragmentation
>> worse.
>>
>> I verified this with DiRT Rally where amdgpu_bo_fault_reserve_notify
>> was much higher in a CPU profiler than anything else in the kernel.
>
>
> Good to know that my expectations on this are correct.
>
> How about fixing the need for contiguous buffers when CPU mapping them?
>
> That should actually be pretty easy to do.
>
>> 2) Monitoring via Gallium HUD
>>
>> We need to expose 2 kernel counters via the INFO ioctl and display
>> those via Gallium HUD:
>> - The number of VRAM CPU page faults. (the number of calls to
>> amdgpu_bo_fault_reserve_notify).
>> - The number of bytes moved by ttm_bo_validate inside
>> amdgpu_bo_fault_reserve_notify.
>>
>> This will help us observe what exactly is happening and fine-tune the
>> throttling when it's done.
>>
>>
>> 3) Solution
>>
>> a) When amdgpu_bo_fault_reserve_notify is called, record the fact.
>> (amdgpu_bo::had_cpu_page_fault = true)
>
>
> What is that good for?
>
>> b) Monitor the MB/s rate at which buffers are moved by
>> amdgpu_bo_fault_reserve_notify. If we get above a specific threshold,
>> don't move the buffer to visible VRAM. Move it to GTT instead. Note
>> that moving to GTT can be cheaper, because moving to visible VRAM is
>> likely to evict a lot of buffers there and unmap them from the CPU,
>> but moving to GTT shouldn't evict or unmap anything.
>
>
> Yeah, had that idea as well. I've been working on adding a context to TTMs
> BO validation call chain.
>
> This way we could add a byte limit on how much TTM will try to evict before
> returning -ENOMEM (or better ENOSPC).
>
>> c) When we get into the CS ioctl and a buffer has had_cpu_page_fault,
>> it can be moved to VRAM if:
>> - the GTT->VRAM move rate is low enough to allow it (this is the
>> existing throttling mechanism)
>> - the visible VRAM move rate is low enough that we will be OK with
>> another CPU page fault if it happens.
>
>
> Interesting idea, need to think a bit about it.
>
> But I would say this has second priority, fixing the contiguous buffer
> requirement should be first. Going to work on that next.

Interesting. I didn't know the contiguous setting wasn't required.

Marek