<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">Hi Jacob,<br>
      <br>
      well first of all we can't rely on the correct behavior of Vulkan.
      In the kernel you have to always assume that userspace is not
      doing the right thing.<br>
      <br>
      <blockquote type="cite">If there is any other UMD applications,
        such as OCL, and OCL driver doesn’t update the SPM_VMID before
        enable SPM, there is VMFault anyway even Vulkan driver restore
        the SPM_VMID to 0.</blockquote>
      <br>
      Question is where with which VMID is this VM fault triggered? The
      SPM VMID?<br>
      <br>
      If yes than that would be quite problematic if an application
      could trigger a VM fault for VMID 0.<br>
      <br>
      Regards,<br>
      Christian.<br>
      <br>
      <br>
      Am 25.02.20 um 08:20 schrieb He, Jacob:<br>
    </div>
    <blockquote type="cite" cite="mid:MN2PR12MB33767BD7E9FFEEAEBE0E14CE9BED0@MN2PR12MB3376.namprd12.prod.outlook.com">
      
      <meta name="Generator" content="Microsoft Word 15 (filtered
        medium)">
      <!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
      <style><!--
/* Font Definitions */
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:DengXian;
        panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
        {font-family:"\@DengXian";
        panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
        {font-family:"Bookman Old Style";
        panose-1:2 5 6 4 5 5 5 2 2 4;}
@font-face
        {font-family:"Bookman Old Style \,serif";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
pre
        {mso-style-priority:99;
        mso-style-link:"HTML Preformatted Char";
        margin:0in;
        margin-bottom:.0001pt;
        font-size:10.0pt;
        font-family:"Courier New";
        color:black;}
span.HTMLPreformattedChar
        {mso-style-name:"HTML Preformatted Char";
        mso-style-priority:99;
        mso-style-link:"HTML Preformatted";
        font-family:"Courier New";
        color:black;}
p.xmsonormal, li.xmsonormal, div.xmsonormal
        {mso-style-name:x_msonormal;
        margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;
        color:black;}
p.xmsipheader4d0fcdd7, li.xmsipheader4d0fcdd7, div.xmsipheader4d0fcdd7
        {mso-style-name:x_msipheader4d0fcdd7;
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;
        color:black;}
p.msipheader4d0fcdd7, li.msipheader4d0fcdd7, div.msipheader4d0fcdd7
        {mso-style-name:msipheader4d0fcdd7;
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;
        color:black;}
.MsoChpDefault
        {mso-style-type:export-only;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.25in 1.0in 1.25in;}
div.WordSection1
        {page:WordSection1;}
--></style>
      <p style="font-family:Arial;font-size:10pt;color:#0078D7;margin:15pt;" align="Left">
        [AMD Official Use Only - Internal Distribution Only]<br>
      </p>
      <br>
      <div>
        <div class="WordSection1">
          <p class="MsoNormal">Vulkan UMD makes sure that the SPM_VMID
            will be updated if the application enables the SPM. So there
            is no this kind of problem for Vulkan application. It’s kind
            of “atomic” operation to enable SPM and update SPM_VMID.
            Vulkan application even is not aware of SPM_VMID.</p>
          <p class="MsoNormal">If there is any other UMD applications,
            such as OCL, and OCL driver doesn’t update the SPM_VMID
            before enable SPM, there is VMFault anyway even Vulkan
            driver restore the SPM_VMID to 0.</p>
          <p class="MsoNormal"><o:p> </o:p></p>
          <p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Bookman Old
              Style",serif">Thanks<o:p></o:p></span></p>
          <p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Bookman Old
              Style",serif">Jacob<o:p></o:p></span></p>
          <p class="MsoNormal"><o:p> </o:p></p>
          <div style="mso-element:para-border-div;border:none;border-top:solid
            #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
            <p class="MsoNormal" style="border:none;padding:0in"><b>From:
              </b><a href="mailto:Christian.Koenig@amd.com" moz-do-not-send="true">Koenig, Christian</a><br>
              <b>Sent: </b>Tuesday, February 25, 2020 2:51 AM<br>
              <b>To: </b><a href="mailto:Jacob.He@amd.com" moz-do-not-send="true">He, Jacob</a>; <a href="mailto:David1.Zhou@amd.com" moz-do-not-send="true">
                Zhou, David(ChunMing)</a>; <a href="mailto:Alexander.Deucher@amd.com" moz-do-not-send="true">Deucher, Alexander</a>;
              <a href="mailto:ckoenig.leichtzumerken@gmail.com" moz-do-not-send="true">Christian König</a>; <a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">
                amd-gfx@lists.freedesktop.org</a><br>
              <b>Subject: </b>Re: [PATCH] drm/amdgpu: Add a chunk ID
              for spm trace</p>
          </div>
          <p class="MsoNormal"><o:p> </o:p></p>
          <p class="MsoNormal"><span style="font-size:10.0pt;color:black">Leave the old
              spm_vmid there is not a problem because other new spm
              enabled job will update it before it’s running and other
              spm disabled job will not access spm_vmid register.</span><span style="color:black"><o:p></o:p></span></p>
          <p class="MsoNormal"><span style="color:black">Do you have a
              high level description of how SPM works? For example would
              it be possible to cause a problem in another application
              if we leave the SPM VMID here and the VMID is reused?<br>
              <br>
              Keep in mind that we need to guarantee process isolation,
              e.g. we can't allow anything bad to happen to another
              application if somebody is stupid enough to turn on SPM
              tracing without setting it up properly.<br>
              <br>
              Regards,<br>
              Christian.<br>
              <br>
              Am 24.02.20 um 09:06 schrieb He, Jacob:<o:p></o:p></span></p>
          <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
            <p class="MsoNormal"><span style="font-size:10.0pt;color:black">Ok, I’m glad that
                there is no uapi change needed.</span><span style="color:black"><o:p></o:p></span></p>
            <p class="MsoNormal"><span style="font-size:10.0pt;color:black">I checked the git
                log, and reserve_vmid was added for shader debugger, not
                gpa. I’m fine to re-use it since the spm will not
                enabled for shader debug in general. I’ll try to change
                my patch.</span><span style="color:black"><o:p></o:p></span></p>
            <p class="MsoNormal"><span style="font-size:10.0pt;color:black">BTW,
                “ring->funcs->setup_spm(ring, NULL)” to clear the
                vmid is not gonna work since the job with spm enabled
                execution is not done yet. Leave the old spm_vmid there
                is not a problem because other new spm enabled job will
                update it before it’s running and other spm disabled job
                will not access spm_vmid register.</span><span style="color:black"><o:p></o:p></span></p>
            <p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
            <p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Bookman Old
                Style \,serif";color:black">Thanks</span><span style="color:black"><o:p></o:p></span></p>
            <p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Bookman Old
                Style \,serif";color:black">Jacob</span><span style="color:black"><o:p></o:p></span></p>
            <p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
            <div style="border:none;border-top:solid #E1E1E1
              1.0pt;padding:3.0pt 0in 0in 0in">
              <p class="MsoNormal"><b><span style="color:black">From: </span></b><a href="mailto:David1.Zhou@amd.com" moz-do-not-send="true">Zhou, David(ChunMing)</a><span style="color:black"><br>
                  <b>Sent: </b>Saturday, February 22, 2020 12:45 AM<br>
                  <b>To: </b></span><a href="mailto:Christian.Koenig@amd.com" moz-do-not-send="true">Koenig, Christian</a><span style="color:black">;
                </span><a href="mailto:Alexander.Deucher@amd.com" moz-do-not-send="true">Deucher, Alexander</a><span style="color:black">;
                </span><a href="mailto:ckoenig.leichtzumerken@gmail.com" moz-do-not-send="true">Christian König</a><span style="color:black">;
                </span><a href="mailto:Jacob.He@amd.com" moz-do-not-send="true">He, Jacob</a><span style="color:black">;
                </span><a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">amd-gfx@lists.freedesktop.org</a><span style="color:black"><br>
                  <b>Subject: </b>RE: [PATCH] drm/amdgpu: Add a chunk
                  ID for spm trace<o:p></o:p></span></p>
            </div>
            <p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
            <p class="msipheader4d0fcdd7" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#0078D7">[AMD
                Official Use Only - Internal Distribution Only]</span></p>
            <p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
            <p class="MsoNormal"><span style="color:black">That’s fine
                to me.<o:p></o:p></span></p>
            <p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
            <p class="MsoNormal"><span style="color:black">-David<o:p></o:p></span></p>
            <p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
            <div>
              <div style="border:none;border-top:solid #E1E1E1
                1.0pt;padding:3.0pt 0in 0in 0in">
                <p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Koenig, Christian
                  </span><a href="mailto:Christian.Koenig@amd.com" moz-do-not-send="true"><Christian.Koenig@amd.com></a><span style="color:black">
                    <br>
                    <b>Sent:</b> Friday, February 21, 2020 11:33 PM<br>
                    <b>To:</b> Deucher, Alexander </span><a href="mailto:Alexander.Deucher@amd.com" moz-do-not-send="true"><Alexander.Deucher@amd.com></a><span style="color:black">; Christian König
                  </span><a href="mailto:ckoenig.leichtzumerken@gmail.com" moz-do-not-send="true"><ckoenig.leichtzumerken@gmail.com></a><span style="color:black">; Zhou, David(ChunMing)
                  </span><a href="mailto:David1.Zhou@amd.com" moz-do-not-send="true"><David1.Zhou@amd.com></a><span style="color:black">; He, Jacob
                  </span><a href="mailto:Jacob.He@amd.com" moz-do-not-send="true"><Jacob.He@amd.com></a><span style="color:black">;
                  </span><a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">amd-gfx@lists.freedesktop.org</a><span style="color:black"><br>
                    <b>Subject:</b> Re: [PATCH] drm/amdgpu: Add a chunk
                    ID for spm trace<o:p></o:p></span></p>
              </div>
            </div>
            <p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
            <div>
              <p class="MsoNormal"><span style="color:black">I would
                  just do this as part of the vm_flush() callback on the
                  ring.<br>
                  <br>
                  E.g. check if the VMID you want to flush is reserved
                  and if yes enable SPM.<br>
                  <br>
                  Maybe pass along a flag or something in the job to
                  make things easier.<br>
                  <br>
                  Christian.<br>
                  <br>
                  Am 21.02.20 um 16:31 schrieb Deucher, Alexander:<o:p></o:p></span></p>
            </div>
            <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
              <p style="margin:15.0pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#317100">[AMD
                  Public Use]</span></p>
              <p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
              <div>
                <div>
                  <p class="MsoNormal"><span style="font-size:12.0pt;color:black">We already
                      have the RESERVE_VMID ioctl interface, can't we
                      just use that internally in the kernel to update
                      the rlc register via the ring when we schedule the
                      relevant IB?  E.g., add a new ring callback to set
                      SPM state and then set it to the reserved vmid
                      before we schedule the ib, and then reset it to 0
                      after the IB in amdgpu_ib_schedule().</span><span style="color:black"><o:p></o:p></span></p>
                </div>
                <div>
                  <p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
                </div>
                <div>
                  <p class="MsoNormal"><span style="font-size:12.0pt;color:black">diff --git
                      a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
                      b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c</span><span style="color:black"><o:p></o:p></span></p>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">index
                        4b2342d11520..e0db9362c6ee 100644</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">---
                        a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">+++
                        b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">@@ -185,6
                        +185,9 @@ int amdgpu_ib_schedule(struct
                        amdgpu_ring *ring, unsigned num_ibs,</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">        if
                        (ring->funcs->insert_start)</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">           
                            ring->funcs->insert_start(ring);</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">+       if
                        (ring->funcs->setup_spm)</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">+          
                            ring->funcs->setup_spm(ring, job);</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">+</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">        if
                        (job) {</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">           
                            r = amdgpu_vm_flush(ring, job,
                        need_pipe_sync);</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">           
                            if (r) {</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">@@ -273,6
                        +276,9 @@ int amdgpu_ib_schedule(struct
                        amdgpu_ring *ring, unsigned num_ibs,</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">           
                            return r;</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">        }</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">+       if
                        (ring->funcs->setup_spm)</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">+          
                            ring->funcs->setup_spm(ring, NULL);</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">+</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">        if
                        (ring->funcs->insert_end)</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black">           
                            ring->funcs->insert_end(ring);</span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <div>
                    <p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
                  </div>
                  <p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
                </div>
                <div>
                  <p class="MsoNormal"><span style="font-size:12.0pt;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
                </div>
                <div>
                  <p class="MsoNormal"><span style="font-size:12.0pt;color:black">Alex</span><span style="color:black"><o:p></o:p></span></p>
                </div>
                <p class="MsoNormal" style="text-align:center" align="center"><span style="color:black"><img style="width:3.8402in;height:.0069in" id="Horizontal_x0020_Line_x0020_1" src="cid:part19.8CC96328.2AB25326@amd.com" class="" width="369" height="1" border="0"></span><span style="color:black"><o:p></o:p></span></p>
                <div id="divRplyFwdMsg">
                  <p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> amd-gfx
                    </span><a href="mailto:amd-gfx-bounces@lists.freedesktop.org" moz-do-not-send="true"><amd-gfx-bounces@lists.freedesktop.org></a><span style="color:black"> on behalf of Christian König
                    </span><a href="mailto:ckoenig.leichtzumerken@gmail.com" moz-do-not-send="true"><ckoenig.leichtzumerken@gmail.com></a><span style="color:black"><br>
                      <b>Sent:</b> Friday, February 21, 2020 5:28 AM<br>
                      <b>To:</b> Zhou, David(ChunMing) </span><a href="mailto:David1.Zhou@amd.com" moz-do-not-send="true"><David1.Zhou@amd.com></a><span style="color:black">; He, Jacob
                    </span><a href="mailto:Jacob.He@amd.com" moz-do-not-send="true"><Jacob.He@amd.com></a><span style="color:black">; Koenig, Christian
                    </span><a href="mailto:Christian.Koenig@amd.com" moz-do-not-send="true"><Christian.Koenig@amd.com></a><span style="color:black">;
                    </span><a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">amd-gfx@lists.freedesktop.org</a><span style="color:black">
                    </span><a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true"><amd-gfx@lists.freedesktop.org></a><span style="color:black"><br>
                      <b>Subject:</b> Re: [PATCH] drm/amdgpu: Add a
                      chunk ID for spm trace <o:p></o:p></span></p>
                  <div>
                    <p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
                  </div>
                </div>
                <div>
                  <div>
                    <p class="MsoNormal" style="background:white"><span style="color:black">That would probably be a
                        no-go, but we could enhance the kernel driver to
                        update the RLC_SPM_VMID register with the
                        reserved VMID.<br>
                        <br>
                        Handling that in userspace is most likely not
                        working anyway, since the RLC registers are
                        usually not accessible by userspace.<br>
                        <br>
                        Regards,<br>
                        Christian.<br>
                        <br>
                        Am 20.02.20 um 16:15 schrieb Zhou,
                        David(ChunMing):<o:p></o:p></span></p>
                  </div>
                  <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
                    <div>
                      <p class="xmsipheader4d0fcdd7" style="margin:0in;margin-bottom:.0001pt;background:white">
                        <span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#0078D7">[AMD
                          Official Use Only - Internal Distribution
                          Only]</span></p>
                      <p class="xmsonormal" style="background:white"> </p>
                      <p class="xmsonormal" style="background:white">You
                        can enhance amdgpu_vm_ioctl In amdgpu_vm.c to
                        return vmid to userspace.
                      </p>
                      <p class="xmsonormal" style="background:white"> </p>
                      <p class="xmsonormal" style="background:white">-David</p>
                      <p class="xmsonormal" style="background:white"> </p>
                      <p class="xmsonormal" style="background:white"> </p>
                      <div>
                        <div style="border:none;border-top:solid #E1E1E1
                          1.0pt;padding:3.0pt 0in 0in 0in">
                          <p class="xmsonormal" style="background:white"><b>From:</b>
                            He, Jacob <a href="mailto:Jacob.He@amd.com" moz-do-not-send="true">
                              <Jacob.He@amd.com></a> <br>
                            <b>Sent:</b> Thursday, February 20, 2020
                            10:46 PM<br>
                            <b>To:</b> Zhou, David(ChunMing) <a href="mailto:David1.Zhou@amd.com" moz-do-not-send="true"><David1.Zhou@amd.com></a>;
                            Koenig, Christian
                            <a href="mailto:Christian.Koenig@amd.com" moz-do-not-send="true"><Christian.Koenig@amd.com></a>;
                            <a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">
                              amd-gfx@lists.freedesktop.org</a><br>
                            <b>Subject:</b> RE: [PATCH] drm/amdgpu: Add
                            a chunk ID for spm trace</p>
                        </div>
                      </div>
                      <p class="xmsonormal" style="background:white"> </p>
                      <p class="xmsonormal" style="background:white">amdgpu_vm_reserve_vmid
                        doesn’t return the reserved vmid back to user
                        space. There is no chance for user mode driver
                        to update RLC_SPM_VMID.</p>
                      <p class="xmsonormal" style="background:white"> </p>
                      <p class="xmsonormal" style="background:white">Thanks</p>
                      <p class="xmsonormal" style="background:white">Jacob</p>
                      <p class="xmsonormal" style="background:white"> </p>
                      <div style="border:none;border-top:solid #E1E1E1
                        1.0pt;padding:3.0pt 0in 0in 0in">
                        <p class="xmsonormal" style="background:white"><b>From:
                          </b><a href="mailto:Jacob.He@amd.com" moz-do-not-send="true">He, Jacob</a><br>
                          <b>Sent: </b>Thursday, February 20, 2020 6:20
                          PM<br>
                          <b>To: </b><a href="mailto:David1.Zhou@amd.com" moz-do-not-send="true">Zhou, David(ChunMing)</a>;
                          <a href="mailto:Christian.Koenig@amd.com" moz-do-not-send="true">
                            Koenig, Christian</a>; <a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">amd-gfx@lists.freedesktop.org</a><br>
                          <b>Subject: </b>RE: [PATCH] drm/amdgpu: Add a
                          chunk ID for spm trace</p>
                      </div>
                      <p class="xmsonormal" style="background:white"> </p>
                      <p class="xmsonormal" style="background:white">Looks
                        like amdgpu_vm_reserve_vmid could work, let me
                        have a try to update the RLC_SPM_VMID with pm4
                        packets in UMD.</p>
                      <p class="xmsonormal" style="background:white"> </p>
                      <p class="xmsonormal" style="background:white">Thanks</p>
                      <p class="xmsonormal" style="background:white">Jacob</p>
                      <p class="xmsonormal" style="background:white"> </p>
                      <div style="border:none;border-top:solid #E1E1E1
                        1.0pt;padding:3.0pt 0in 0in 0in">
                        <p class="xmsonormal" style="background:white"><b>From:
                          </b><a href="mailto:David1.Zhou@amd.com" moz-do-not-send="true">Zhou, David(ChunMing)</a><br>
                          <b>Sent: </b>Thursday, February 20, 2020
                          10:13 AM<br>
                          <b>To: </b><a href="mailto:Christian.Koenig@amd.com" moz-do-not-send="true">Koenig, Christian</a>;
                          <a href="mailto:Jacob.He@amd.com" moz-do-not-send="true">
                            He, Jacob</a>; <a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">amd-gfx@lists.freedesktop.org</a><br>
                          <b>Subject: </b>RE: [PATCH] drm/amdgpu: Add a
                          chunk ID for spm trace</p>
                      </div>
                      <p class="xmsonormal" style="background:white"> </p>
                      <p class="xmsonormal" style="background:white">[AMD
                        Official Use Only - Internal Distribution Only]<br>
                        <br>
                        Christian is right here, that will cause many
                        problems for simply using VMID in kernel.<br>
                        We already have an pair interface for RGP, I
                        think you can use it instead of involving
                        additional kernel change.<br>
                        amdgpu_vm_reserve_vmid/
                        amdgpu_vm_unreserve_vmid.<br>
                        <br>
                        -David<br>
                        <br>
                        -----Original Message-----<br>
                        From: amd-gfx <<a href="mailto:amd-gfx-bounces@lists.freedesktop.org" moz-do-not-send="true">amd-gfx-bounces@lists.freedesktop.org</a>>
                        On Behalf Of Christian König<br>
                        Sent: Wednesday, February 19, 2020 7:03 PM<br>
                        To: He, Jacob <<a href="mailto:Jacob.He@amd.com" moz-do-not-send="true">Jacob.He@amd.com</a>>;
                        <a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">
                          amd-gfx@lists.freedesktop.org</a><br>
                        Subject: Re: [PATCH] drm/amdgpu: Add a chunk ID
                        for spm trace<br>
                        <br>
                        Am 19.02.20 um 11:15 schrieb Jacob He:<br>
                        > [WHY]<br>
                        > When SPM trace enabled, SPM_VMID should be
                        updated with the current <br>
                        > vmid.<br>
                        ><br>
                        > [HOW]<br>
                        > Add a chunk id, AMDGPU_CHUNK_ID_SPM_TRACE,
                        so that UMD can tell us <br>
                        > which job should update SPM_VMID.<br>
                        > Right before a job is submitted to GPU, set
                        the SPM_VMID accordingly.<br>
                        ><br>
                        > [Limitation]<br>
                        > Running more than one SPM trace enabled
                        processes simultaneously is <br>
                        > not supported.<br>
                        <br>
                        Well there are multiple problems with that
                        patch.<br>
                        <br>
                        First of all you need to better describe what
                        SPM tracing is in the commit message.<br>
                        <br>
                        Then the updating of mmRLC_SPM_MC_CNTL must be
                        executed asynchronously on the ring. Otherwise
                        we might corrupt an already executing SPM trace.<br>
                        <br>
                        And you also need to make sure to disable the
                        tracing again or otherwise we run into a bunch
                        of trouble when the VMID is reused.<br>
                        <br>
                        You also need to make sure that IBs using the
                        SPM trace are serialized with each other, e.g.
                        hack into amdgpu_ids.c file and make sure that
                        only one VMID at a time can have that attribute.<br>
                        <br>
                        Regards,<br>
                        Christian.<br>
                        <br>
                        ><br>
                        > Change-Id:
                        Ic932ef6ac9dbf244f03aaee90550e8ff3a675666<br>
                        > Signed-off-by: Jacob He <<a href="mailto:jacob.he@amd.com" moz-do-not-send="true">jacob.he@amd.com</a>><br>
                        > ---<br>
                        >   drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c 
                        |  7 +++++++<br>
                        >   drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c  |
                        10 +++++++---<br>
                        >   drivers/gpu/drm/amd/amdgpu/amdgpu_job.h
                        |  1 +<br>
                        >   drivers/gpu/drm/amd/amdgpu/amdgpu_rlc.h
                        |  1 +<br>
                        >   drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c  |
                        15 ++++++++++++++-<br>
                        >   drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c  
                        |  3 ++-<br>
                        >   drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c  
                        |  3 ++-<br>
                        >   drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c   |
                        15 ++++++++++++++-<br>
                        >   8 files changed, 48 insertions(+), 7
                        deletions(-)<br>
                        ><br>
                        > diff --git
                        a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c <br>
                        > b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c<br>
                        > index f9fa6e104fef..3f32c4db5232 100644<br>
                        > ---
                        a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c<br>
                        > +++
                        b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c<br>
                        > @@ -113,6 +113,7 @@ static int
                        amdgpu_cs_parser_init(struct amdgpu_cs_parser
                        *p, union drm_amdgpu_cs<br>
                        >        uint32_t uf_offset = 0;<br>
                        >        int i;<br>
                        >        int ret;<br>
                        > +     bool update_spm_vmid = false;<br>
                        >   <br>
                        >        if (cs->in.num_chunks == 0)<br>
                        >                return 0;<br>
                        > @@ -221,6 +222,10 @@ static int
                        amdgpu_cs_parser_init(struct amdgpu_cs_parser
                        *p, union drm_amdgpu_cs<br>
                        >                case
                        AMDGPU_CHUNK_ID_SYNCOBJ_TIMELINE_SIGNAL:<br>
                        >                        break;<br>
                        >   <br>
                        > +             case
                        AMDGPU_CHUNK_ID_SPM_TRACE:<br>
                        > +                     update_spm_vmid =
                        true;<br>
                        > +                     break;<br>
                        > +<br>
                        >                default:<br>
                        >                        ret = -EINVAL;<br>
                        >                        goto
                        free_partial_kdata;<br>
                        > @@ -231,6 +236,8 @@ static int
                        amdgpu_cs_parser_init(struct amdgpu_cs_parser
                        *p, union drm_amdgpu_cs<br>
                        >        if (ret)<br>
                        >                goto free_all_kdata;<br>
                        >   <br>
                        > +     p->job->need_update_spm_vmid =
                        update_spm_vmid;<br>
                        > +<br>
                        >        if (p->ctx->vram_lost_counter
                        != p->job->vram_lost_counter) {<br>
                        >                ret = -ECANCELED;<br>
                        >                goto free_all_kdata;<br>
                        > diff --git
                        a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c <br>
                        > b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c<br>
                        > index cae81914c821..36faab12b585 100644<br>
                        > ---
                        a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c<br>
                        > +++
                        b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c<br>
                        > @@ -156,9 +156,13 @@ int
                        amdgpu_ib_schedule(struct amdgpu_ring *ring,
                        unsigned num_ibs,<br>
                        >                return -EINVAL;<br>
                        >        }<br>
                        >   <br>
                        > -     if (vm && !job->vmid) {<br>
                        > -             dev_err(adev->dev, "VM IB
                        without ID\n");<br>
                        > -             return -EINVAL;<br>
                        > +     if (vm) {<br>
                        > +             if (!job->vmid) {<br>
                        > +                     dev_err(adev->dev,
                        "VM IB without ID\n");<br>
                        > +                     return -EINVAL;<br>
                        > +             } else if
                        (adev->gfx.rlc.funcs->update_spm_vmid
                        && job->need_update_spm_vmid) {<br>
                        > +                    
                        adev->gfx.rlc.funcs->update_spm_vmid(adev,
                        job->vmid);<br>
                        > +             }<br>
                        >        }<br>
                        >   <br>
                        >        alloc_size =
                        ring->funcs->emit_frame_size + num_ibs *
                        diff --git <br>
                        > a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h <br>
                        > b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h<br>
                        > index 2e2110dddb76..4582536961c7 100644<br>
                        > ---
                        a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h<br>
                        > +++
                        b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.h<br>
                        > @@ -52,6 +52,7 @@ struct amdgpu_job {<br>
                        >        bool                   
                        vm_needs_flush;<br>
                        >        uint64_t                vm_pd_addr;<br>
                        >        unsigned                vmid;<br>
                        > +     bool                   
                        need_update_spm_vmid;<br>
                        >        unsigned                pasid;<br>
                        >        uint32_t                gds_base,
                        gds_size;<br>
                        >        uint32_t                gws_base,
                        gws_size;<br>
                        > diff --git
                        a/drivers/gpu/drm/amd/amdgpu/amdgpu_rlc.h <br>
                        > b/drivers/gpu/drm/amd/amdgpu/amdgpu_rlc.h<br>
                        > index d3d4707f2168..52509c254cbd 100644<br>
                        > ---
                        a/drivers/gpu/drm/amd/amdgpu/amdgpu_rlc.h<br>
                        > +++
                        b/drivers/gpu/drm/amd/amdgpu/amdgpu_rlc.h<br>
                        > @@ -126,6 +126,7 @@ struct amdgpu_rlc_funcs
                        {<br>
                        >        void (*stop)(struct amdgpu_device
                        *adev);<br>
                        >        void (*reset)(struct amdgpu_device
                        *adev);<br>
                        >        void (*start)(struct amdgpu_device
                        *adev);<br>
                        > +     void (*update_spm_vmid)(struct
                        amdgpu_device *adev, unsigned vmid);<br>
                        >   };<br>
                        >   <br>
                        >   struct amdgpu_rlc {<br>
                        > diff --git
                        a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c <br>
                        > b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c<br>
                        > index 5e9fb0976c6c..91eb788d6229 100644<br>
                        > ---
                        a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c<br>
                        > +++
                        b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c<br>
                        > @@ -4214,6 +4214,18 @@ static int
                        gfx_v10_0_update_gfx_clock_gating(struct
                        amdgpu_device *adev,<br>
                        >        return 0;<br>
                        >   }<br>
                        >   <br>
                        > +static void
                        gfx_v10_0_update_spm_vmid(struct amdgpu_device
                        *adev, <br>
                        > +unsigned vmid) {<br>
                        > +     u32 data;<br>
                        > +<br>
                        > +     data = RREG32_SOC15(GC, 0,
                        mmRLC_SPM_MC_CNTL);<br>
                        > +<br>
                        > +     data &=
                        ~RLC_SPM_MC_CNTL__RLC_SPM_VMID_MASK;<br>
                        > +     data |= (vmid &
                        RLC_SPM_MC_CNTL__RLC_SPM_VMID_MASK) << <br>
                        > +RLC_SPM_MC_CNTL__RLC_SPM_VMID__SHIFT;<br>
                        > +<br>
                        > +     WREG32_SOC15(GC, 0,
                        mmRLC_SPM_MC_CNTL, data); }<br>
                        > +<br>
                        >   static const struct amdgpu_rlc_funcs
                        gfx_v10_0_rlc_funcs = {<br>
                        >        .is_rlc_enabled =
                        gfx_v10_0_is_rlc_enabled,<br>
                        >        .set_safe_mode =
                        gfx_v10_0_set_safe_mode, @@ -4224,7 +4236,8 @@ <br>
                        > static const struct amdgpu_rlc_funcs
                        gfx_v10_0_rlc_funcs = {<br>
                        >        .resume = gfx_v10_0_rlc_resume,<br>
                        >        .stop = gfx_v10_0_rlc_stop,<br>
                        >        .reset = gfx_v10_0_rlc_reset,<br>
                        > -     .start = gfx_v10_0_rlc_start<br>
                        > +     .start = gfx_v10_0_rlc_start,<br>
                        > +     .update_spm_vmid =
                        gfx_v10_0_update_spm_vmid<br>
                        >   };<br>
                        >   <br>
                        >   static int
                        gfx_v10_0_set_powergating_state(void *handle,
                        diff --git <br>
                        > a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c <br>
                        > b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c<br>
                        > index 8f20a5dd44fe..b24fc55cf13a 100644<br>
                        > --- a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c<br>
                        > +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c<br>
                        > @@ -4221,7 +4221,8 @@ static const struct
                        amdgpu_rlc_funcs gfx_v7_0_rlc_funcs = {<br>
                        >        .resume = gfx_v7_0_rlc_resume,<br>
                        >        .stop = gfx_v7_0_rlc_stop,<br>
                        >        .reset = gfx_v7_0_rlc_reset,<br>
                        > -     .start = gfx_v7_0_rlc_start<br>
                        > +     .start = gfx_v7_0_rlc_start,<br>
                        > +     .update_spm_vmid = NULL<br>
                        >   };<br>
                        >   <br>
                        >   static int gfx_v7_0_early_init(void
                        *handle) diff --git <br>
                        > a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c <br>
                        > b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c<br>
                        > index fa245973de12..66640d2b6b37 100644<br>
                        > --- a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c<br>
                        > +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c<br>
                        > @@ -5600,7 +5600,8 @@ static const struct
                        amdgpu_rlc_funcs iceland_rlc_funcs = {<br>
                        >        .resume = gfx_v8_0_rlc_resume,<br>
                        >        .stop = gfx_v8_0_rlc_stop,<br>
                        >        .reset = gfx_v8_0_rlc_reset,<br>
                        > -     .start = gfx_v8_0_rlc_start<br>
                        > +     .start = gfx_v8_0_rlc_start,<br>
                        > +     .update_spm_vmid = NULL<br>
                        >   };<br>
                        >   <br>
                        >   static void
                        gfx_v8_0_update_medium_grain_clock_gating(struct
                        <br>
                        > amdgpu_device *adev, diff --git <br>
                        > a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c <br>
                        > b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c<br>
                        > index 9b7ff783e9a5..df872f949f68 100644<br>
                        > --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c<br>
                        > +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c<br>
                        > @@ -4704,6 +4704,18 @@ static int
                        gfx_v9_0_update_gfx_clock_gating(struct
                        amdgpu_device *adev,<br>
                        >        return 0;<br>
                        >   }<br>
                        >   <br>
                        > +static void
                        gfx_v9_0_update_spm_vmid(struct amdgpu_device
                        *adev, <br>
                        > +unsigned vmid) {<br>
                        > +     u32 data;<br>
                        > +<br>
                        > +     data = RREG32_SOC15(GC, 0,
                        mmRLC_SPM_MC_CNTL);<br>
                        > +<br>
                        > +     data &=
                        ~RLC_SPM_MC_CNTL__RLC_SPM_VMID_MASK;<br>
                        > +     data |= (vmid &
                        RLC_SPM_MC_CNTL__RLC_SPM_VMID_MASK) << <br>
                        > +RLC_SPM_MC_CNTL__RLC_SPM_VMID__SHIFT;<br>
                        > +<br>
                        > +     WREG32_SOC15(GC, 0,
                        mmRLC_SPM_MC_CNTL, data); }<br>
                        > +<br>
                        >   static const struct amdgpu_rlc_funcs
                        gfx_v9_0_rlc_funcs = {<br>
                        >        .is_rlc_enabled =
                        gfx_v9_0_is_rlc_enabled,<br>
                        >        .set_safe_mode =
                        gfx_v9_0_set_safe_mode, @@ -4715,7 +4727,8 @@ <br>
                        > static const struct amdgpu_rlc_funcs
                        gfx_v9_0_rlc_funcs = {<br>
                        >        .resume = gfx_v9_0_rlc_resume,<br>
                        >        .stop = gfx_v9_0_rlc_stop,<br>
                        >        .reset = gfx_v9_0_rlc_reset,<br>
                        > -     .start = gfx_v9_0_rlc_start<br>
                        > +     .start = gfx_v9_0_rlc_start,<br>
                        > +     .update_spm_vmid =
                        gfx_v9_0_update_spm_vmid<br>
                        >   };<br>
                        >   <br>
                        >   static int
                        gfx_v9_0_set_powergating_state(void *handle,<br>
                        <br>
                        _______________________________________________<br>
                        amd-gfx mailing list<br>
                        <a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">amd-gfx@lists.freedesktop.org</a><br>
                        <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Calexander.deucher%40amd.com%7Ce68811adba0647c651a308d7b6b8d4c5%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637178777321057824&sdata=uRP0wQyfVKk0QoGlWM36nbeOY67nIjfD6vou2A%2BybEc%3D&reserved=0" moz-do-not-send="true">https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=02%7C01%7Cdavid1.zhou%40amd.com%7C354be34ff18e4f424f6708d7b52b43b0%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637177069753914395&amp;sdata=9rSL4kgPJweuZ4EJpdqtqTxyCVGEkmsg6aUzbtvGFrs%3D&amp;reserved=0</a></p>
                      <p class="xmsonormal" style="background:white"> </p>
                      <p class="xmsonormal" style="background:white"> </p>
                    </div>
                    <p class="MsoNormal" style="margin-bottom:12.0pt;background:white"><span style="color:black"> <o:p></o:p></span></p>
                    <pre style="background:white">_______________________________________________</pre>
                    <pre style="background:white">amd-gfx mailing list</pre>
                    <pre style="background:white"><a href="mailto:amd-gfx@lists.freedesktop.org" moz-do-not-send="true">amd-gfx@lists.freedesktop.org</a></pre>
                    <pre style="background:white"><a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Calexander.deucher%40amd.com%7Ce68811adba0647c651a308d7b6b8d4c5%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637178777321067823&sdata=7o2j5dTvK%2B5Gihwd5IB%2Blp1Bi0ZOx%2B%2F%2Bmi8wpRh4NBs%3D&reserved=0" moz-do-not-send="true">https://lists.freedesktop.org/mailman/listinfo/amd-gfx</a></pre>
                  </blockquote>
                  <p class="MsoNormal" style="background:white"><span style="color:black"> <o:p></o:p></span></p>
                </div>
              </div>
            </blockquote>
            <p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
            <p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
          </blockquote>
          <p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
          <p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>