<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi Rico,<br>
<br>
maybe it is a good idea to look into the git history and/or google
the mailing list history a bit more.<br>
<br>
I do briefly remember that we disabled freeing up the stolen VGA
memory on some hardware generations because somebody was accessing
that memory even after VGA was turned off.<br>
<br>
At that time we couldn't figure out what was going wrong, but it
is perfectly possible that those memory training fixes your are
working on here are related to that issue.<br>
<br>
On the other hand if you can't find anything immediately feel free
to go with Alex suggestion, if something goes wrong we can still
revert the change.<br>
<br>
Regards,<br>
Christian.<br>
<br>
Am 09.01.20 um 04:15 schrieb Yin, Tianci (Rico):<br>
</div>
<blockquote type="cite"
cite="mid:MN2PR12MB2957F23A709E07E481AD630B95390@MN2PR12MB2957.namprd12.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
<p
style="font-family:Arial;font-size:10pt;color:#0078D7;margin:15pt;"
align="Left">
[AMD Official Use Only - Internal Distribution Only]<br>
</p>
<br>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Ok, thanks very much Alex!<br>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b> Alex
Deucher <a class="moz-txt-link-rfc2396E" href="mailto:alexdeucher@gmail.com"><alexdeucher@gmail.com></a><br>
<b>Sent:</b> Thursday, January 9, 2020 11:12<br>
<b>To:</b> Yin, Tianci (Rico) <a class="moz-txt-link-rfc2396E" href="mailto:Tianci.Yin@amd.com"><Tianci.Yin@amd.com></a><br>
<b>Cc:</b> Christian König
<a class="moz-txt-link-rfc2396E" href="mailto:ckoenig.leichtzumerken@gmail.com"><ckoenig.leichtzumerken@gmail.com></a>; Koenig, Christian
<a class="moz-txt-link-rfc2396E" href="mailto:Christian.Koenig@amd.com"><Christian.Koenig@amd.com></a>; Long, Gang
<a class="moz-txt-link-rfc2396E" href="mailto:Gang.Long@amd.com"><Gang.Long@amd.com></a>; Wang, Kevin(Yang)
<a class="moz-txt-link-rfc2396E" href="mailto:Kevin1.Wang@amd.com"><Kevin1.Wang@amd.com></a>; Xu, Feifei
<a class="moz-txt-link-rfc2396E" href="mailto:Feifei.Xu@amd.com"><Feifei.Xu@amd.com></a>; amd-gfx list
<a class="moz-txt-link-rfc2396E" href="mailto:amd-gfx@lists.freedesktop.org"><amd-gfx@lists.freedesktop.org></a>; Tuikov, Luben
<a class="moz-txt-link-rfc2396E" href="mailto:Luben.Tuikov@amd.com"><Luben.Tuikov@amd.com></a>; Deucher, Alexander
<a class="moz-txt-link-rfc2396E" href="mailto:Alexander.Deucher@amd.com"><Alexander.Deucher@amd.com></a>; Yuan, Xiaojie
<a class="moz-txt-link-rfc2396E" href="mailto:Xiaojie.Yuan@amd.com"><Xiaojie.Yuan@amd.com></a>; Zhang, Hawking
<a class="moz-txt-link-rfc2396E" href="mailto:Hawking.Zhang@amd.com"><Hawking.Zhang@amd.com></a><br>
<b>Subject:</b> Re: [PATCH] drm/amdgpu: fix modprobe failure
of the 2nd GPU when GDDR6 training enabled</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span
style="font-size:11pt;">
<div class="PlainText">On Wed, Jan 8, 2020 at 10:07 PM
Yin, Tianci (Rico) <a class="moz-txt-link-rfc2396E" href="mailto:Tianci.Yin@amd.com"><Tianci.Yin@amd.com></a> wrote:<br>
><br>
> [AMD Official Use Only - Internal Distribution
Only]<br>
><br>
><br>
> Thanks Alex and Christian!<br>
><br>
> Hi Christian,<br>
><br>
> On ASICs with gmc v10, I find amdgpu_bo_late_init()
is not invoked, so stolen memory never get free, on
other ASICs with gmc v9/v8/v7/v6, stolen memory was
freed in gmc_v*_0_late_init(). I don't know why, are
there some special reasons or just missed by coding?<br>
><br>
<br>
Looks like it should be added. Possibly got lost when
we merged the<br>
navi code from the topic branch.<br>
<br>
Alex<br>
<br>
> Thanks!<br>
><br>
> Rico<br>
><br>
> ________________________________<br>
> From: Christian König
<a class="moz-txt-link-rfc2396E" href="mailto:ckoenig.leichtzumerken@gmail.com"><ckoenig.leichtzumerken@gmail.com></a><br>
> Sent: Wednesday, January 8, 2020 23:04<br>
> To: Alex Deucher <a class="moz-txt-link-rfc2396E" href="mailto:alexdeucher@gmail.com"><alexdeucher@gmail.com></a>;
Koenig, Christian <a class="moz-txt-link-rfc2396E" href="mailto:Christian.Koenig@amd.com"><Christian.Koenig@amd.com></a><br>
> Cc: Long, Gang <a class="moz-txt-link-rfc2396E" href="mailto:Gang.Long@amd.com"><Gang.Long@amd.com></a>; Wang,
Kevin(Yang) <a class="moz-txt-link-rfc2396E" href="mailto:Kevin1.Wang@amd.com"><Kevin1.Wang@amd.com></a>; Xu, Feifei
<a class="moz-txt-link-rfc2396E" href="mailto:Feifei.Xu@amd.com"><Feifei.Xu@amd.com></a>; Yin, Tianci (Rico)
<a class="moz-txt-link-rfc2396E" href="mailto:Tianci.Yin@amd.com"><Tianci.Yin@amd.com></a>; amd-gfx list
<a class="moz-txt-link-rfc2396E" href="mailto:amd-gfx@lists.freedesktop.org"><amd-gfx@lists.freedesktop.org></a>; Tuikov, Luben
<a class="moz-txt-link-rfc2396E" href="mailto:Luben.Tuikov@amd.com"><Luben.Tuikov@amd.com></a>; Deucher, Alexander
<a class="moz-txt-link-rfc2396E" href="mailto:Alexander.Deucher@amd.com"><Alexander.Deucher@amd.com></a>; Yuan, Xiaojie
<a class="moz-txt-link-rfc2396E" href="mailto:Xiaojie.Yuan@amd.com"><Xiaojie.Yuan@amd.com></a>; Zhang, Hawking
<a class="moz-txt-link-rfc2396E" href="mailto:Hawking.Zhang@amd.com"><Hawking.Zhang@amd.com></a><br>
> Subject: Re: [PATCH] drm/amdgpu: fix modprobe
failure of the 2nd GPU when GDDR6 training enabled<br>
><br>
> Am 08.01.20 um 15:49 schrieb Alex Deucher:<br>
> > On Wed, Jan 8, 2020 at 7:52 AM Christian König
<a class="moz-txt-link-rfc2396E" href="mailto:christian.koenig@amd.com"><christian.koenig@amd.com></a> wrote:<br>
> >> Am 08.01.20 um 13:36 schrieb Tianci Yin:<br>
> >>> From: "Tianci.Yin"
<a class="moz-txt-link-rfc2396E" href="mailto:tianci.yin@amd.com"><tianci.yin@amd.com></a><br>
> >>><br>
> >>> [why]<br>
> >>> In dual GPUs scenario, stolen_size is
assigned to zero on the 2nd GPU,<br>
> >>> then the bottom region of VRAM was
allocated as GTT, unfortunately<br>
> >>> a small region of bottom VRAM was
encroached by UMC firmware during<br>
> >>> GDDR6 BIST training, this cause
pagefault.<br>
> >> What I'm missing here is why is the stolen
size zero on the 2nd GPU?<br>
> >><br>
> >> Maybe we need to read the stolen size
after posting the GPU instead?<br>
> > There is no stolen memory on secondary GPUs
because there is no pre-OS<br>
> > console using that memory.<br>
><br>
> Ah! Yeah that makes sense.<br>
><br>
> But in this case I would say we should change the
patch like the following:<br>
><br>
> adev->gmc.stolen_size =
min(adev->gmc.stolen_size,<br>
> AMDGPU_STOLEN_VGA_DEFAULT_SIZE);<br>
><br>
> And in amdgpu_ttm_late_init():<br>
><br>
> /* Can't free the stolen VGA memory when it might
be used for memory<br>
> training again. */<br>
> if (!adev->fw_vram_usage.mem_train_support)<br>
> amdgpu_bo_free_kernel(....<br>
><br>
><br>
> Regards,<br>
> Christian.<br>
><br>
><br>
> ><br>
> > Alex<br>
> ><br>
> >> Regards,<br>
> >> Christian.<br>
> >><br>
> >>> [how]<br>
> >>> Forcing stolen_size to 3MB, then the
bottom region of VRAM was<br>
> >>> allocated as stolen memory, GTT
corruption avoid.<br>
> >>> The stolen memory of the 2nd GPU will
be free in late_init phase,<br>
> >>> no memory wasted.<br>
> >>><br>
> >>> Change-Id:
Icd0ad7de41333282949bb1e3e676c6c307ddd081<br>
> >>> Signed-off-by: Tianci.Yin
<a class="moz-txt-link-rfc2396E" href="mailto:tianci.yin@amd.com"><tianci.yin@amd.com></a><br>
> >>> ---<br>
> >>>
drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h | 6 ++++++<br>
> >>>
drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 21
+++++++++++++++++++++<br>
> >>> 2 files changed, 27 insertions(+)<br>
> >>><br>
> >>> diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h<br>
> >>> index c91dd602d5f1..440b793316df
100644<br>
> >>> ---
a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h<br>
> >>> +++
b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h<br>
> >>> @@ -60,6 +60,11 @@<br>
> >>> */<br>
> >>> #define AMDGPU_GMC_FAULT_TIMEOUT
5000ULL<br>
> >>><br>
> >>> +/*<br>
> >>> + * Default stolen memory size, 1024 *
768 * 4<br>
> >>> + */<br>
> >>> +#define
AMDGPU_STOLEN_VGA_DEFAULT_SIZE 0x300000<br>
> >>> +<br>
> >>> struct firmware;<br>
> >>><br>
> >>> /*<br>
> >>> @@ -192,6 +197,7 @@ struct amdgpu_gmc
{<br>
> >>> uint32_t
srbm_soft_reset;<br>
> >>> bool
prt_warning;<br>
> >>> uint64_t
stolen_size;<br>
> >>> + bool
stolen_temp_reserved;<br>
> >>> /* apertures */<br>
> >>> u64
shared_aperture_start;<br>
> >>> u64
shared_aperture_end;<br>
> >>> diff --git
a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c<br>
> >>> index 7dc8c068c62a..0c96b67d6ca7
100644<br>
> >>> ---
a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c<br>
> >>> +++
b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c<br>
> >>> @@ -566,6 +566,11 @@ static int
gmc_v10_0_late_init(void *handle)<br>
> >>> struct amdgpu_device *adev =
(struct amdgpu_device *)handle;<br>
> >>> int r;<br>
> >>><br>
> >>> + if
(adev->gmc.stolen_temp_reserved) {<br>
> >>> +
amdgpu_bo_late_init(adev);<br>
> >>> +
adev->gmc.stolen_temp_reserved = false;<br>
> >>> + }<br>
> >>> +<br>
> >>> r =
amdgpu_gmc_allocate_vm_inv_eng(adev);<br>
> >>> if (r)<br>
> >>> return r;<br>
> >>> @@ -756,6 +761,22 @@ static int
gmc_v10_0_sw_init(void *handle)<br>
> >>> return r;<br>
> >>><br>
> >>> adev->gmc.stolen_size =
gmc_v10_0_get_vbios_fb_size(adev);<br>
> >>> + /*<br>
> >>> + * In dual GPUs scenario,
stolen_size is assigned to zero on the 2nd GPU,<br>
> >>> + * then the bottom region of
VRAM was allocated as GTT, unfortunately<br>
> >>> + * a small region of bottom VRAM
was encroached by UMC firmware during<br>
> >>> + * GDDR6 BIST training, this
cause pagefault.<br>
> >>> + * The page fault can be fixed
by forcing stolen_size to 3MB, then the bottom<br>
> >>> + * region of VRAM was allocated
as stolen memory, GTT corruption avoid.<br>
> >>> + * The stolen memory of the 2nd
GPU will be free in late_init phase,<br>
> >>> + * no memory wasted.<br>
> >>> + */<br>
> >>> + if
(adev->fw_vram_usage.mem_train_support &&<br>
> >>> + adev->gmc.stolen_size
== 0) {<br>
> >>> + adev->gmc.stolen_size
= AMDGPU_STOLEN_VGA_DEFAULT_SIZE;<br>
> >>> +
adev->gmc.stolen_temp_reserved = true;<br>
> >>> + } else<br>
> >>> +
adev->gmc.stolen_temp_reserved = false;<br>
> >>><br>
> >>> /* Memory manager */<br>
> >>> r = amdgpu_bo_init(adev);<br>
> >>
_______________________________________________<br>
> >> amd-gfx mailing list<br>
> >> <a class="moz-txt-link-abbreviated" href="mailto:amd-gfx@lists.freedesktop.org">amd-gfx@lists.freedesktop.org</a><br>
> >> <a
href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7CTianci.Yin%40amd.com%7Cb9b1622ed60e4ab36c6408d794b1d75e%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637141363862418597&sdata=YT8zKlbLX0XdzqcLrZQaV6sKFWXS5nQTNMAMT9BMK70%3D&reserved=0"
moz-do-not-send="true">
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7CTianci.Yin%40amd.com%7Cb9b1622ed60e4ab36c6408d794b1d75e%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637141363862418597&sdata=YT8zKlbLX0XdzqcLrZQaV6sKFWXS5nQTNMAMT9BMK70%3D&reserved=0</a><br>
> >
_______________________________________________<br>
> > amd-gfx mailing list<br>
> > <a class="moz-txt-link-abbreviated" href="mailto:amd-gfx@lists.freedesktop.org">amd-gfx@lists.freedesktop.org</a><br>
> > <a
href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7CTianci.Yin%40amd.com%7Cb9b1622ed60e4ab36c6408d794b1d75e%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637141363862428589&sdata=LLVRzFBxFKqTltpvsK%2F2l9CwnlUFzFKmDoPPHdO5e1I%3D&reserved=0"
moz-do-not-send="true">
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7CTianci.Yin%40amd.com%7Cb9b1622ed60e4ab36c6408d794b1d75e%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637141363862428589&sdata=LLVRzFBxFKqTltpvsK%2F2l9CwnlUFzFKmDoPPHdO5e1I%3D&reserved=0</a><br>
><br>
</div>
</span></font></div>
</div>
</blockquote>
<br>
</body>
</html>