[Intel-xe] FW: [PATCH v9 0/3] PAT and cache coherency support

[Matthew Auld]

On 08/11/2023 11:30, Tomasz Mistat wrote:
> compute-umd_uapi_review_team at intel.com,bartosz.dunajski at intel.com,saurabhg.gupta at intel.com,ulisses.furquim at intel.com,michal.mrozek at intel.com,tomasz.mistat at intel.com,piotr.rozenfeld at intel.com,ayaz.siddiqui at intel.com,luis.strano at intel.com,filip.hazubski at intel.com,filip.hazubski at intel.com,mateusz.naklicki at intel.com,kamil.kopryk at intel.com
> Bcc:
> Subject: Re: [Intel-xe] [PATCH v9 0/3] PAT and cache coherency support
> Reply-To:
> In-Reply-To: <20231103154309.402159-5-matthew.auld at intel.com>
> 
> On 2023-11-03 at 15:43:10 +0000, Matthew Auld wrote:
>> Branch available here:
>> https://gitlab.freedesktop.org/mwa/kernel/-/tree/xe-pat-index?ref_type=heads
>>
>> IGT changes:
>> https://patchwork.freedesktop.org/series/124667/
>>
>> Mesa:
>> https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25462
>>
>> Goal here is to allow userspace to directly control the pat_index when mapping
>> memory via the ppGTT, in addtion to the CPU caching mode. This is very much
>> needed on newer igpu platforms which allow incoherent GT access, where the
>> choice over the cache level and expected coherency is best left to userspace
>> depending on their usecase.  In the future there may also be other stuff encoded
>> in the pat_index, so giving userspace direct control will also be needed there.
>>
>> To support this we added new gem_create uAPI for selecting the CPU cache
>> mode to use for system memory, including the expected GPU coherency mode. There
>> are various restrictions here for the selected coherency mode and compatible CPU
>> cache modes.  With that in place the actual pat_index can now be provided as
>> part of vm_bind. The only restriction is that the coherency mode of the
>> pat_index must be at least as coherent as the gem_create coherency mode. There
>> are also some special cases like with userptr and dma-buf.
>>
>> v2:
>>    - Loads of improvements/tweaks. Main changes are to now allow
>>      gem_create.coh_mode <= coh_mode(pat_index), rather than it needing to match
>>      exactly. This simplifies the dma-buf policy from userspace pov. Also we now
>>      only consider COH_NONE and COH_AT_LEAST_1WAY.
>> v3:
>>    - Rebase. Split the pte_encode() refactoring, plus various smaller tweaks and
>>      fixes.
>> v4:
>>    - Rebase on Lucas' new series.
>>    - Drop UC cache mode.
>>    - s/smem_cpu_caching/cpu_caching/. Idea is to make VRAM WC explicit in the
>>      uapi, plus make it more future proof.
>> v5:
>>    - Rebase, plus some small tweaks and fixes.
>> v6:
>>    - CI hooks fixes + checkpatch.
>> v7:
>>    - Some small tweaks
>> v8:
>>    - Rebase on Xe2 PAT table additions.
>> v9:
>>    - Drop coh_mode.
>>
>> -- 
>> 2.41.0
>>
> 
> I'm sharing down here questions passed by bartosz.dunajski at intel.com
> lets continue review here at respective ML
> 
> Sent: Wednesday, November 8, 2023 09:05
> Subject: RE: [Intel-xe] [PATCH v9 0/3] PAT and cache coherency support
> 
> Just to confirm my understanding. We will have 2 separate APIs?
> 1.	For gem_create to explicitly set coherency/caching according to API enums

Yeah, you just need to provide the cpu_caching mode (wc or wb). If you 
later mmap() the object this same caching mode is used. Also any KMD 
internal things like swap-in will also respect this caching mode. The 
coh_mode is now removed.

> 2.	For vm_bind to set PAT index that is aligned with Bspec and controls more than #1. It can also override coherency/caching that was previously set?

Right, as part of vm_bind you provide whatever PAT index control (as per 
Bspec) you want for each mapping of the object (you could map it 
multiple times).

When choosing a PAT index the cpu_caching only matters if you choose an 
incoherent PAT index (coh_none in Bspec), since we don't want to allow 
cpu_caching=wb and coh_none, since that leads to some complications in 
the KMD.

> 
>