<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:等线;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
@font-face
{font-family:微软雅黑;
panose-1:2 11 5 3 2 2 4 2 2 4;}
@font-face
{font-family:"\@微软雅黑";}
@font-face
{font-family:"\@等线";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
font-size:11.0pt;
font-family:"Arial",sans-serif;
mso-ligatures:standardcontextual;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Arial",sans-serif;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Arial",sans-serif;
color:blue;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 99.75pt 1.0in 99.75pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<p style="font-family:Calibri;font-size:10pt;color:#0000FF;margin:5pt;font-style:normal;font-weight:normal;text-decoration:none;" align="Left">
[AMD Official Use Only - AMD Internal Distribution Only]<br>
</p>
<br>
<div>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:blue">Ping
<a id="OWAAMF23B6D94AD8B4D71B0DEF392AB1F1B65" href="mailto:Lijo.Lazar@amd.com"><span style="font-family:"Arial",sans-serif;text-decoration:none">@Lazar, Lijo</span></a>,
<a id="OWAAMB3028D499C26463B90271DAAEA9C55AB" href="mailto:Christian.Koenig@amd.com">
<span style="font-family:"Arial",sans-serif;text-decoration:none">@Koenig, Christian</span></a>…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:blue"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:blue">Kindly pls review the updated patch in advance and we can discuss your suggestions in tomorrow's meeting. Thanks for your great support.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:blue"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:blue;mso-ligatures:standardcontextual"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:blue;mso-ligatures:standardcontextual">Rgds/Owen<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Arial",sans-serif;color:blue"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Deng, Emily <Emily.Deng@amd.com>
<br>
<b>Sent:</b> Monday, May 26, 2025 9:56 AM<br>
<b>To:</b> Zhang, GuoQing (Sam) <GuoQing.Zhang@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; Lazar, Lijo <Lijo.Lazar@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com><br>
<b>Cc:</b> Zhao, Victor <Victor.Zhao@amd.com>; Chang, HaiJun <HaiJun.Chang@amd.com>; Zhang, GuoQing (Sam) <GuoQing.Zhang@amd.com>; Zhang, Owen(SRDC) <Owen.Zhang2@amd.com>; Ma, Qing (Mark) <Qing.Ma@amd.com>; amd-gfx@lists.freedesktop.org<br>
<b>Subject:</b> RE: [PATCH v8 0/4] enable xgmi node migration support for hibernate on SRIOV<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin:5.0pt"><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:blue">[AMD Official Use Only - AMD Internal Distribution Only]<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoPlainText"><a id="OWAAM93342BC5185947079C46533D3A83FAD6" href="mailto:Christian.Koenig@amd.com"><span style="font-family:"Arial",sans-serif;text-decoration:none">@Koenig, Christian</span></a> and
<a id="OWAAMF29C1B0CDD8544DA820ECD26D0382CA2" href="mailto:Lijo.Lazar@amd.com"><span style="font-family:"Arial",sans-serif;text-decoration:none">@Lazar, Lijo</span></a><o:p></o:p></p>
<p class="MsoPlainText">Could you help review these changes again?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Best whishes<o:p></o:p></p>
<p class="MsoPlainText">Emily Deng<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">>-----Original Message-----<o:p></o:p></p>
<p class="MsoPlainText">>From: Samuel Zhang <<a href="mailto:guoqing.zhang@amd.com">guoqing.zhang@amd.com</a>><o:p></o:p></p>
<p class="MsoPlainText">>Sent: Thursday, May 22, 2025 6:41 PM<o:p></o:p></p>
<p class="MsoPlainText">>To: <a href="mailto:amd-gfx@lists.freedesktop.org">amd-gfx@lists.freedesktop.org</a><o:p></o:p></p>
<p class="MsoPlainText">>Cc: Zhao, Victor <<a href="mailto:Victor.Zhao@amd.com">Victor.Zhao@amd.com</a>>; Chang, HaiJun<o:p></o:p></p>
<p class="MsoPlainText">><<a href="mailto:HaiJun.Chang@amd.com">HaiJun.Chang@amd.com</a>>; Zhang, GuoQing (Sam) <<a href="mailto:GuoQing.Zhang@amd.com">GuoQing.Zhang@amd.com</a>>;<o:p></o:p></p>
<p class="MsoPlainText">>Koenig, Christian <<a href="mailto:Christian.Koenig@amd.com">Christian.Koenig@amd.com</a>>; Deucher, Alexander<o:p></o:p></p>
<p class="MsoPlainText">><<a href="mailto:Alexander.Deucher@amd.com">Alexander.Deucher@amd.com</a>>; Zhang, Owen(SRDC) <<a href="mailto:Owen.Zhang2@amd.com">Owen.Zhang2@amd.com</a>>;<o:p></o:p></p>
<p class="MsoPlainText">>Ma, Qing (Mark) <<a href="mailto:Qing.Ma@amd.com">Qing.Ma@amd.com</a>>; Lazar, Lijo <<a href="mailto:Lijo.Lazar@amd.com">Lijo.Lazar@amd.com</a>>; Deng,<o:p></o:p></p>
<p class="MsoPlainText">>Emily <<a href="mailto:Emily.Deng@amd.com">Emily.Deng@amd.com</a>><o:p></o:p></p>
<p class="MsoPlainText">>Subject: [PATCH v8 0/4] enable xgmi node migration support for hibernate on SRIOV<o:p></o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">>On SRIOV and VM environment, customer may need to switch to new vGPU indexes<o:p></o:p></p>
<p class="MsoPlainText">>after hibernate and then resume the VM. For GPUs with XGMI, `vram_start` will<o:p></o:p></p>
<p class="MsoPlainText">>change in this case, the FB aperture gpu address of VRAM BOs will also change.<o:p></o:p></p>
<p class="MsoPlainText">>These gpu addresses need to be updated when resume. But these addresses are all<o:p></o:p></p>
<p class="MsoPlainText">>over the KMD codebase, updating each of them is error-prone and not acceptable.<o:p></o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">>The solution is to use pdb0 page table to cover both vram and gart memory and use<o:p></o:p></p>
<p class="MsoPlainText">>pdb0 virtual gpu address instead. When gpu indexes change, the virtual gpu address<o:p></o:p></p>
<p class="MsoPlainText">>won't change.<o:p></o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">>For psp and smu, pdb0's gpu address does not work, so the original FB aperture gpu<o:p></o:p></p>
<p class="MsoPlainText">>address is used instead. They need to be updated when resume with changed<o:p></o:p></p>
<p class="MsoPlainText">>vGPUs.<o:p></o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">>v2:<o:p></o:p></p>
<p class="MsoPlainText">>- remove physical_node_id_changed<o:p></o:p></p>
<p class="MsoPlainText">>- set vram_start to 0 to switch cached gpu addr to gart aperture<o:p></o:p></p>
<p class="MsoPlainText">>- cleanup pdb0 patch<o:p></o:p></p>
<p class="MsoPlainText">>v3:<o:p></o:p></p>
<p class="MsoPlainText">>- remove gmc_v9_0_init_sw_mem_ranges() call<o:p></o:p></p>
<p class="MsoPlainText">>- remove vram_offset memeber<o:p></o:p></p>
<p class="MsoPlainText">>- add 4 refactoring patch to remove cached gpu addr<o:p></o:p></p>
<p class="MsoPlainText">>- cleanup pdb0 patch<o:p></o:p></p>
<p class="MsoPlainText">>v4:<o:p></o:p></p>
<p class="MsoPlainText">>- remove gmc_v9_0_mc_init() call and `refresh` update.<o:p></o:p></p>
<p class="MsoPlainText">>- do not set `fb_start` in mmhub_v1_8_get_fb_location() when pdb0 enabled.<o:p></o:p></p>
<p class="MsoPlainText">>v5:<o:p></o:p></p>
<p class="MsoPlainText">>- add amdgpu_virt_xgmi_migrate_enabled() check<o:p></o:p></p>
<p class="MsoPlainText">>- move vram_base_offset update to pdb0 patch<o:p></o:p></p>
<p class="MsoPlainText">>- remove 4 refactoring patches to remove cached gpu addr<o:p></o:p></p>
<p class="MsoPlainText">>- add patch to fix IH not working issue when resume with new VF<o:p></o:p></p>
<p class="MsoPlainText">>v6: per Lijo feedback<o:p></o:p></p>
<p class="MsoPlainText">>- rename amdgpu_device_update_xgmi_info() to amdgpu_virt_resume()<o:p></o:p></p>
<p class="MsoPlainText">>- merge xgmi node and vram_base_offset update, IH fix into amdgpu_virt_resume()<o:p></o:p></p>
<p class="MsoPlainText">>- remove 2 unnecessary gpu addr update changes<o:p></o:p></p>
<p class="MsoPlainText">>v7: per Christian feedback<o:p></o:p></p>
<p class="MsoPlainText">>- remove pdb0_enabled and add gmc_v9_0_is_pdb0_enabled()<o:p></o:p></p>
<p class="MsoPlainText">>- remove amdgpu_gmc_vram_location() call in amdgpu_gmc_sysvm_location()<o:p></o:p></p>
<p class="MsoPlainText">>- remove check in mmhub_v1_8_get_fb_location() and update fb_start/fb_end on<o:p></o:p></p>
<p class="MsoPlainText">>resume<o:p></o:p></p>
<p class="MsoPlainText">>v8:<o:p></o:p></p>
<p class="MsoPlainText">>- use cached fb_start in amdgpu_bo_fb_aper_addr()<o:p></o:p></p>
<p class="MsoPlainText">>- remove fb_start/fb_end update in amdgpu_virt_resume() and<o:p></o:p></p>
<p class="MsoPlainText">>amdgpu_gmc_sysvm_location()<o:p></o:p></p>
<p class="MsoPlainText">>- use vram_start to set regVM_CONTEXT0_PAGE_TABLE_START_ADDR_*<o:p></o:p></p>
<p class="MsoPlainText">>- move check to the callsite of amdgpu_virt_resume()<o:p></o:p></p>
<p class="MsoPlainText">>- add gmc.xgmi.node_segment_size check in amdgpu_virt_xgmi_migrate_enabled()<o:p></o:p></p>
<p class="MsoPlainText">>- rename gmc_v9_0_is_pdb0_enabled() to amdgpu_gmc_is_pdb0_enabled()<o:p></o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">>Samuel Zhang (4):<o:p></o:p></p>
<p class="MsoPlainText">> drm/amdgpu: update xgmi info and vram_base_offset on resume<o:p></o:p></p>
<p class="MsoPlainText">> drm/amdgpu: update GPU addresses for SMU and PSP<o:p></o:p></p>
<p class="MsoPlainText">> drm/amdgpu: enable pdb0 for hibernation on SRIOV<o:p></o:p></p>
<p class="MsoPlainText">> drm/amdgpu: fix fence fallback timer expired error<o:p></o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 38 ++++++++++++++++++++++<o:p></o:p></p>
<p class="MsoPlainText">> drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 28 ++++++++++++----<o:p></o:p></p>
<p class="MsoPlainText">> drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 2 +-<o:p></o:p></p>
<p class="MsoPlainText">> drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h | 1 +<o:p></o:p></p>
<p class="MsoPlainText">> drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 20 ++++++++++++<o:p></o:p></p>
<p class="MsoPlainText">>drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 1 +<o:p></o:p></p>
<p class="MsoPlainText">> drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 23 +++++++++++++<o:p></o:p></p>
<p class="MsoPlainText">> drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c | 3 ++<o:p></o:p></p>
<p class="MsoPlainText">> drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h | 7 ++++<o:p></o:p></p>
<p class="MsoPlainText">> drivers/gpu/drm/amd/amdgpu/gfxhub_v1_2.c | 8 +++--<o:p></o:p></p>
<p class="MsoPlainText">> drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 13 +++++---<o:p></o:p></p>
<p class="MsoPlainText">> drivers/gpu/drm/amd/amdgpu/mmhub_v1_8.c | 6 ++--<o:p></o:p></p>
<p class="MsoPlainText">> drivers/gpu/drm/amd/pm/swsmu/amdgpu_smu.c | 18 ++++++++++<o:p></o:p></p>
<p class="MsoPlainText">> 13 files changed, 151 insertions(+), 17 deletions(-)<o:p></o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">>--<o:p></o:p></p>
<p class="MsoPlainText">>2.43.5<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>