[PATCH] drm/amdgpu: fix modprobe failure of the 2nd GPU when GDDR6 training enabled

Alex Deucher alexdeucher at gmail.com
Thu Jan 9 03:12:49 UTC 2020 

On Wed, Jan 8, 2020 at 10:07 PM Yin, Tianci (Rico) <Tianci.Yin at amd.com> wrote:
>
> [AMD Official Use Only - Internal Distribution Only]
>
>
> Thanks Alex and Christian!
>
> Hi Christian,
>
> On ASICs with gmc v10, I find amdgpu_bo_late_init() is not invoked, so stolen memory never get free, on other ASICs with gmc v9/v8/v7/v6, stolen memory was freed in gmc_v*_0_late_init(). I don't know why, are there some special reasons or just missed by coding?
>

Looks like it should be added.  Possibly got lost when we merged the
navi code from the topic branch.

Alex

> Thanks!
>
> Rico
>
> ________________________________
> From: Christian König <ckoenig.leichtzumerken at gmail.com>
> Sent: Wednesday, January 8, 2020 23:04
> To: Alex Deucher <alexdeucher at gmail.com>; Koenig, Christian <Christian.Koenig at amd.com>
> Cc: Long, Gang <Gang.Long at amd.com>; Wang, Kevin(Yang) <Kevin1.Wang at amd.com>; Xu, Feifei <Feifei.Xu at amd.com>; Yin, Tianci (Rico) <Tianci.Yin at amd.com>; amd-gfx list <amd-gfx at lists.freedesktop.org>; Tuikov, Luben <Luben.Tuikov at amd.com>; Deucher, Alexander <Alexander.Deucher at amd.com>; Yuan, Xiaojie <Xiaojie.Yuan at amd.com>; Zhang, Hawking <Hawking.Zhang at amd.com>
> Subject: Re: [PATCH] drm/amdgpu: fix modprobe failure of the 2nd GPU when GDDR6 training enabled
>
> Am 08.01.20 um 15:49 schrieb Alex Deucher:
> > On Wed, Jan 8, 2020 at 7:52 AM Christian König <christian.koenig at amd.com> wrote:
> >> Am 08.01.20 um 13:36 schrieb Tianci Yin:
> >>> From: "Tianci.Yin" <tianci.yin at amd.com>
> >>>
> >>> [why]
> >>> In dual GPUs scenario, stolen_size is assigned to zero on the 2nd GPU,
> >>> then the bottom region of VRAM was allocated as GTT, unfortunately
> >>> a small region of bottom VRAM was encroached by UMC firmware during
> >>> GDDR6 BIST training, this cause pagefault.
> >> What I'm missing here is why is the stolen size zero on the 2nd GPU?
> >>
> >> Maybe we need to read the stolen size after posting the GPU instead?
> > There is no stolen memory on secondary GPUs because there is no pre-OS
> > console using that memory.
>
> Ah! Yeah that makes sense.
>
> But in this case I would say we should change the patch like the following:
>
> adev->gmc.stolen_size = min(adev->gmc.stolen_size,
> AMDGPU_STOLEN_VGA_DEFAULT_SIZE);
>
> And in amdgpu_ttm_late_init():
>
> /* Can't free the stolen VGA memory when it might be used for memory
> training again. */
> if (!adev->fw_vram_usage.mem_train_support)
>      amdgpu_bo_free_kernel(....
>
>
> Regards,
> Christian.
>
>
> >
> > Alex
> >
> >> Regards,
> >> Christian.
> >>
> >>> [how]
> >>> Forcing stolen_size to 3MB, then the bottom region of VRAM was
> >>> allocated as stolen memory, GTT corruption avoid.
> >>> The stolen memory of the 2nd GPU will be free in late_init phase,
> >>> no memory wasted.
> >>>
> >>> Change-Id: Icd0ad7de41333282949bb1e3e676c6c307ddd081
> >>> Signed-off-by: Tianci.Yin <tianci.yin at amd.com>
> >>> ---
> >>>    drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h |  6 ++++++
> >>>    drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c  | 21 +++++++++++++++++++++
> >>>    2 files changed, 27 insertions(+)
> >>>
> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> >>> index c91dd602d5f1..440b793316df 100644
> >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h
> >>> @@ -60,6 +60,11 @@
> >>>     */
> >>>    #define AMDGPU_GMC_FAULT_TIMEOUT    5000ULL
> >>>
> >>> +/*
> >>> + * Default stolen memory size, 1024 * 768 * 4
> >>> + */
> >>> +#define AMDGPU_STOLEN_VGA_DEFAULT_SIZE       0x300000
> >>> +
> >>>    struct firmware;
> >>>
> >>>    /*
> >>> @@ -192,6 +197,7 @@ struct amdgpu_gmc {
> >>>        uint32_t                srbm_soft_reset;
> >>>        bool                    prt_warning;
> >>>        uint64_t                stolen_size;
> >>> +     bool                    stolen_temp_reserved;
> >>>        /* apertures */
> >>>        u64                     shared_aperture_start;
> >>>        u64                     shared_aperture_end;
> >>> diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
> >>> index 7dc8c068c62a..0c96b67d6ca7 100644
> >>> --- a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
> >>> +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
> >>> @@ -566,6 +566,11 @@ static int gmc_v10_0_late_init(void *handle)
> >>>        struct amdgpu_device *adev = (struct amdgpu_device *)handle;
> >>>        int r;
> >>>
> >>> +     if (adev->gmc.stolen_temp_reserved) {
> >>> +             amdgpu_bo_late_init(adev);
> >>> +             adev->gmc.stolen_temp_reserved = false;
> >>> +     }
> >>> +
> >>>        r = amdgpu_gmc_allocate_vm_inv_eng(adev);
> >>>        if (r)
> >>>                return r;
> >>> @@ -756,6 +761,22 @@ static int gmc_v10_0_sw_init(void *handle)
> >>>                return r;
> >>>
> >>>        adev->gmc.stolen_size = gmc_v10_0_get_vbios_fb_size(adev);
> >>> +     /*
> >>> +      * In dual GPUs scenario, stolen_size is assigned to zero on the 2nd GPU,
> >>> +      * then the bottom region of VRAM was allocated as GTT, unfortunately
> >>> +      * a small region of bottom VRAM was encroached by UMC firmware during
> >>> +      * GDDR6 BIST training, this cause pagefault.
> >>> +      * The page fault can be fixed by forcing stolen_size to 3MB, then the bottom
> >>> +      * region of VRAM was allocated as stolen memory, GTT corruption avoid.
> >>> +      * The stolen memory of the 2nd GPU will be free in late_init phase,
> >>> +      * no memory wasted.
> >>> +      */
> >>> +     if (adev->fw_vram_usage.mem_train_support &&
> >>> +             adev->gmc.stolen_size == 0) {
> >>> +             adev->gmc.stolen_size = AMDGPU_STOLEN_VGA_DEFAULT_SIZE;
> >>> +             adev->gmc.stolen_temp_reserved = true;
> >>> +     } else
> >>> +             adev->gmc.stolen_temp_reserved = false;
> >>>
> >>>        /* Memory manager */
> >>>        r = amdgpu_bo_init(adev);
> >> _______________________________________________
> >> amd-gfx mailing list
> >> amd-gfx at lists.freedesktop.org
> >> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Ctianci.yin%40amd.com%7C1365f16aef4c418f0db308d7944c0f26%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637140926723776401&sdata=ubt7GGRaAvLzv%2F%2BUXxaTW%2FgGbKkRKWv6%2BB50fHHW3Xc%3D&reserved=0
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx at lists.freedesktop.org
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Ctianci.yin%40amd.com%7C1365f16aef4c418f0db308d7944c0f26%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637140926723776401&sdata=ubt7GGRaAvLzv%2F%2BUXxaTW%2FgGbKkRKWv6%2BB50fHHW3Xc%3D&reserved=0
>

More information about the amd-gfx mailing list