[Intel-xe] LLC configurating, mmap and bo cache management questions

[Tvrtko Ursulin]

Hi Thomas,

On 06/12/2023 08:26, Hellstrom, Thomas wrote:
> On Tue, 2023-12-05 at 14:19 +0000, Tvrtko Ursulin wrote:
>>
>> Hi,
>>
>> We are working on adding xe support to ChromeOS minigbm and have a
>> couple questions.
>>
>> If I follow things correctly with xe mmap caching mode is fixed to
>> object caching modes set at bo create. For framebuffers it will be WC
>> and for the rest userspace can choose WB or WC via
>> drm_xe_gem_create->cpu_caching. (Unless discrete, when WB cannot be
>> used
>> at all.)
>>
>> AFAICT minigbm basically cares about two transition points. Lets call
>> them CPU access begin and end.
>>
>> 1)
>> When a bo is mmapped it wants to invalidate the cache, which looks to
>> be
>> about making sure all GPU writes have landed to the backing store. In
>> the i915 world that translates to the set_domain ioctl.
>>
>> What is the uapi for this with xe, or it is somehow guaranteed to not
>> be
>> needed?
> 
> Signalling a user-fence or dma-fence obtained as an out-fence from an
> exec call will guarantee GPU caches are flushed. Currently I don't
> think there is anything like gem wait in the uAPI, although Matt is
> just about to add functionality to wait on all outstanding work on an
> exec_queue.

Problem I see is there are no execs or therefore fences in the minigbm 
ABI. It's just buffers, created or imported, CPU access and some other 
stuff.

And it is quite extensively used in the OS so I assume it has to work (I 
mean the invalidation/flushing was not put in there for no reason), in 
other words, where the i915 backend today does DRM_I915_GEM_SET_DOMAIN 
on "cpu access begin", which is buffer based, I am not clear how to 
implement that with xe.

For the outstanding work you mention, since you say it is about 
exec_queue, I assume again it will not work purely with buffers? If so 
it probably won't be useful for minigbm.

Also if I look at all the other minigbm backends, I see a mix of behaviours:

  * msm and vc4 appear to not concern themselves with any of this.

  * rockchip appears to be doing full bounce buffering via memcpy on CPU 
access begin/end.

  * i915 and amdgpu respectively use 
DRM_I915_GEM_SET_DOMAIN/DRM_AMDGPU_GEM_WAIT_IDLE (also buffer based, not 
execution queue). Andgpu curiously does not do any flushing on CPU 
access end.

Digging into git history, both DRM_I915_GEM_SET_DOMAIN on CPU access 
begin and clflushing on end were added to fix various CTS test failures. 
So I guess we could also wait and see what happens there. If those or 
some will be failing with xe too then propose adding some new uapi. Or 
if manual testing will start reporting visual corruption in the UI 
elements or such.

>> 2)
>> When a bo is unmapped, or CPU access finished, it wants to flush the
>> CPU
>> caches. That is /almost/ completely a CPU operation, where it just
>> needs
>> to either clflush or invalidate the WC buffer respectively, if not
>> the
>> fact that clflush can be skipped on platforms with LLC.
>>
>> I did not see an equivalent of an I915_PARAM_HAS_LLC in xe? Did I
>> miss
>> it or what it is the plan for querying this detail?
> 
> XeKMD is generally coherent, except if UMD selects a GPU PAT index with
> limited coherency together with WB instead of WC memory. In that case,
> UMD is responsible for doing the needed CLFLUSH-ing, whereas KMD only
> ensures initial clearing of the pages is CLFLUSHED for security
> reasons.
> 
> I'm not 100% sure if UMD can actually select WB with limited coherency
> PAT index in the initial uAPI revision, but Matthew has received
> requests for that so any additional input here on performance
> implications is appreciated.
> 
> The thinking here is otherwise that GPU PAT indices with limited
> coherency should be used together with WC memory in the same situations
> as VRAM/LMEM is used on DGFX.

Hm okay, this would be the VM BIND side of things which deals with GPU 
PATs. From the CPU side it is just CPU caching modes implicitly selected 
via DRM_XE_GEM_CPU_CACHING_WB/WC.

The question about the analogue of I915_PARAM_HAS_LLC was AFAIU about a 
performance optimisation where UMD is deciding whether it is okay to 
skip issuing clflush for the mapped bo if DRM_XE_GEM_CPU_CACHING_WB was 
used. (If DRM_XE_GEM_CPU_CACHING_WC was used it obviously only needs to 
flush the write combine buffer.)

Looking at what Mesa is doing it appears it is not using 
I915_PARAM_HAS_LLC but has its own device info tables. So I guess 
minigbm xe backend will have to do the same if xe will not provide the 
analogue query.

Regards,

Tvrtko