[Intel-gfx] [PATCH v2 2/2] drm/i915/bxt: Fix inadvertent CPU snooping due to incorrect MOCS config

[Eero Tamminen]

Hi,

On 26.04.2016 20:25, Frederick, Michael T wrote:
> Sorry I'm not tracking all the MOCs discussions.  I just want to indicate what the coherency means in SoC for BXT.
>
> GTI sets the non-inclusive bit on the IDI interface based on how it treats the memory.  In BXT case where there is no uncore cache, "non-inclusive" just indicates snoop or not.  BXT has a snoop filter in order to make the latency of snooping GT from a core roughly similar to snooping another core.
>
> For BXT:
> If GTI sets non-inclusive=0 (i.e. coherent): transaction looks up in the SF and the SA snoops the cores.  The potential impact here is that for high BW coherent traffic, the SF will become the BW limiter of the system and cap BW at 33% * 34GBps. For writes like WCILFs snoops to cores must be resolved before SA requests WR data from GT.  For reads the common case should have no impact because snoop latency is generally much less than memory data latency.  In general snoop latency for a core is relatively small, but there is also the prospect that a core could be down (e.g. ratio change) or loaded w/ snooping.
> If GTI sets non-inclusive=1 (i.e. non-coherent): transaction takes the SF bypass and the SA does not snoop the cores.  This is best for high-BW since it removes the SF bottleneck and doesn't require core interaction.

Thanks for the explanation!

AFAIK:

* In regards to 3D driver operations, CPU side doesn't modify the buffer 
contents while GPU is working on them.  CPU side sets up the buffers 
(textures, VBOs, batches etc), and then (after a flush) GPU is asked to 
act on them.

* For things like texture streaming, the driver either internally 
synchronizes the data or creates a new copy of it whenever application 
tells that data is updated.  There's always some kind of "upload" 
involved (GL API needs it as non-integrated GPU's don't share memory 
with CPU).

While it's possible that there's a case where snooping would be needed, 
I cannot think of any myself.

Daniel, Chris, did you have some concrete example in mind where 3D 
driver would require CPU to snoop GPU?


	- Eero

Disclaimer: I'm not a 3D driver developer myself.

>
> Thanks, Mike
>
>
> -----Original Message-----
> From: Tamminen, Eero T
> Sent: Tuesday, April 26, 2016 10:19 AM
> To: Daniel Vetter <daniel at ffwll.ch>
> Cc: Chris Wilson <chris at chris-wilson.co.uk>; Deak, Imre <imre.deak at intel.com>; intel-gfx at lists.freedesktop.org; Rantala, Valtteri <valtteri.rantala at intel.com>; Frederick, Michael T <michael.t.frederick at intel.com>; Ville Syrjälä <ville.syrjala at linux.intel.com>
> Subject: Re: [Intel-gfx] [PATCH v2 2/2] drm/i915/bxt: Fix inadvertent CPU snooping due to incorrect MOCS config
>
> Hi,
>
> On 26.04.2016 17:30, Daniel Vetter wrote:
>> On Tue, Apr 26, 2016 at 05:26:43PM +0300, Eero Tamminen wrote:
> [...]
>>> What this kernel ABI (index entry #2) has been agreed & documented to
>>> provide?
>>>
>>> I thought this entry is supposed to replace the writeback LLC/eLLC
>>> cache MOCS setting Mesa is using on (e.g. BDW) to speed up accesses
>>> to a memory area which it knows always to be accessed so that it can be cached.
>>>
>>> If app runs on HW where LLC/eLLC is missing, giving the app extra
>>> slowdown instead of potential speedup sounds like failed HW
>>> abstraction. :-)
>>
>> Well mesa needs to know llc vs. !llc anyway to not totally suck,
>
> What do you think it should do with that information?
>
> I assume you to mean, that Mesa needs to know the *amount* of LLC and change its behavior based on that amount, not just whether it's present.
>
> In that case Mesa does, and has always totally "sucked".  Mesa on earlier GEN(s) cached everything that can be cached, and I assume it to try to do that with GEN9 too.
>
>
> However, based on our MOCS testing on BDW, that actually gives the best overall perf results.  On average it doesn't give much, but it was better than any straightforward (buffer size/type) heuristics for making something not to be cached in effort to utilize LLC "better".
>
> It seemed that LLC is too small to have meaningful generic heuristics for normal 3D workloads (or they need to be very complex, something needing months of testing & iteration, or be per application, not generic).
>
> eLLC could be a different matter as it's large enough that one can put e.g. color/depth buffer there.
>
> Skip cache setting for LLC may also be useful, if it works (as it in a sense extends the cache size), and render compression can also change things.  Problem with RBC is that it makes assumptions about memory areas usage even less reliable as you don't know how well the content compresses.
>
>
>> and defining entry #2 as "coherent, always" makes sense. I thought
>> entry 0 was the reaonable default aka pte passthrough and hence managed by kernel?
>>
>> If mesa asks for nonsense, the kernel is happy to oblige.
>
>
> 	- Eero
>