<div dir="ltr"><div>I'll answer two questions asked:</div><div> </div><div>1) SDMA doesn't need GCR at the end of IBs. It's because SDMA writes bypass GL2 and at the same time they invalidate all cache lines they touch.</div><div> </div><div>2)</div><div>> If we always insert a GL2C invalidate at every EOP of every IB from every engine, why we need a GL2C invalidate before IB execute ?</div><div> </div><div>I just sent you a counterexample on a private thread that proves that invalidation in RELEASE_MEM doesn't accomplish anything. The invalidation flag is there because:</div><div>1) it was inherited from gfx9, which was inherited from gfx8, which was inherited from gfx7, which doesn't have the WB flag, so INV has to be used instead.</div><div>2) to hide bugs </div><div> </div><div>Marek </div></div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 29, 2020 at 7:24 AM Liu, Monk <<a href="mailto:Monk.Liu@amd.com">Monk.Liu@amd.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="white" lang="EN-US"> <div class="gmail-m_-2967598555427486213WordSection1"> >> Well from my understanding I think that a G2LC invalidation is still necessary before an IB executes. Agree, I think before an IB executes the only thing we need on GL2C is the invalidation, not the flush . >> The problem is that the memory of the IB could also be cached because of some activity of the GFX or Compute rings. If we always insert a GL2C invalidate at every EOP of every IB from every engine, why we need a GL2C invalidate before IB execute ? <div> _____________________________________ Monk Liu|GPU Virtualization Team |AMD <img style="width: 0.8333in; height: 0.8333in;" id="gmail-m_-2967598555427486213_x0000_i1027" src="cid:171d0f0a62b4cff311" alt="sig-cloud-gpu" width="80" height="80"> </div> <div> <div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in"> From: Koenig, Christian <<a href="mailto:Christian.Koenig@amd.com" target="_blank">Christian.Koenig@amd.com</a>> Sent: Wednesday, April 29, 2020 5:38 PM To: Liu, Monk <<a href="mailto:Monk.Liu@amd.com" target="_blank">Monk.Liu@amd.com</a>>; Marek Olšák <<a href="mailto:maraeo@gmail.com" target="_blank">maraeo@gmail.com</a>>; amd-gfx mailing list <<a href="mailto:amd-gfx@lists.freedesktop.org" target="_blank">amd-gfx@lists.freedesktop.org</a>> Subject: Re: drm/amdgpu: invalidate L2 before SDMA IBs (on gfx10) </div> </div> <div> Well from my understanding I think that a G2LC invalidation is still necessary before an IB executes. The problem is that the memory of the IB could also be cached because of some activity of the GFX or Compute rings. Regards, Christian. Am 29.04.20 um 11:35 schrieb Liu, Monk: </div> <blockquote style="margin-top:5pt;margin-bottom:5pt"> Here is the reason we should always insert a “sync mem” packet at the FENCE place of SDMA, not before IB emit. By always inserting “sync mem” in the FENCE place we can make sure:1 <ol style="margin-top:0in" type="1" start="1"> <li class="gmail-m_-2967598555427486213MsoListParagraph" style="margin-left:0in">data is really flushed to system memory before CPU try to read it </li><li class="gmail-m_-2967598555427486213MsoListParagraph" style="margin-left:0in">all the G2LC is invalidated by “sync mem”, thus in the next round SDMA IB, it won’t get staled data from G2LC cache </li></ol> by inserting “sync mem” in prior to IB could only achieve : Avoid get staled data in g2lc during IB execution for GFX/COMPUTE ring since they have release_mem packet so it is inherently doing the G2LC flush and invalidate upon a fence signaled <div> _____________________________________ Monk Liu|GPU Virtualization Team |AMD <img style="width: 0.8333in; height: 0.8333in;" id="gmail-m_-2967598555427486213_x0000_i1025" src="cid:171d0f0a62b4cff311" alt="sig-cloud-gpu" width="80" height="80"> </div> <div> <div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in"> From: Liu, Monk Sent: Wednesday, April 29, 2020 5:06 PM To: 'Marek Olšák' <a href="mailto:maraeo@gmail.com" target="_blank"><maraeo@gmail.com></a>; amd-gfx mailing list <a href="mailto:amd-gfx@lists.freedesktop.org" target="_blank"><amd-gfx@lists.freedesktop.org></a>; Koenig, Christian <a href="mailto:Christian.Koenig@amd.com" target="_blank"><Christian.Koenig@amd.com></a> Subject: RE: drm/amdgpu: invalidate L2 before SDMA IBs (on gfx10) </div> </div> Hi <a id="gmail-m_-2967598555427486213OWAAM709EE8E0E6054CD48698E878A98E9795" href="mailto:Christian.Koenig@amd.com" target="_blank"> @Koenig, Christian</a> & Marek I still have some concerns regarding Marek’s patch, correct me if I’m wrong See that Marek put a SDMA_OP_GCR_REQ before emitting IB, to make sure SDMA won’t get stale cache data during the IB execution. But that “SDMA_OP_GCR_REQ” only invalidate/flush the GFXHUB’s G2LC cache right ? what if the memory is changed by MM or CPU (out side of GFXHUB) ? Can this “ SDMA_OP_GCR_REQ” force MMHUB or even CPU to flush their operation result from their cache to memory ?? Besides, with my understanding the “EOP” of gfx ring is doing the thing of “invalidate/flush” L2 cache upon a fence signaled, so what we should do on SDMA5 is to insert this “SDMA_OP_GCR_REQ” Right before thee “emit_fence” of SDMA (this is what windows KMD do) thanks _____________________________________ Monk Liu|GPU Virtualization Team |AMD <img style="width: 0.8333in; height: 0.8333in;" id="gmail-m_-2967598555427486213Picture_x0020_1" src="cid:171d0f0a62b4cff311" alt="sig-cloud-gpu" width="80" height="80" border="0"> From: amd-gfx <<a href="mailto:amd-gfx-bounces@lists.freedesktop.org" target="_blank">amd-gfx-bounces@lists.freedesktop.org</a>> On Behalf Of Marek Ol?ák Sent: Saturday, April 25, 2020 4:52 PM To: amd-gfx mailing list <<a href="mailto:amd-gfx@lists.freedesktop.org" target="_blank">amd-gfx@lists.freedesktop.org</a>> Subject: drm/amdgpu: invalidate L2 before SDMA IBs (on gfx10) <div> <div> This should fix SDMA hangs on gfx10. </div> <div> </div> <div> Marek </div> </div> </blockquote> </div> </div> </blockquote></div>