[PATCH 5/8] drm/amdgpu/mes12: add support for setting gang submit

[Alex Deucher]

On Tue, Apr 29, 2025 at 3:16 PM Liu, Shaoyun <Shaoyun.Liu at amd.com> wrote:
>
> [AMD Official Use Only - AMD Internal Distribution Only]
>
>  kiq still be  used for kernel queues map/unmap including the  scheduler queue itself . I think we can keep this design.   The  gang related will go through  SCHED.

What about queue resets?

Alex

>
> Regards
> Shaoyun.liu
>
> -----Original Message-----
> From: Alex Deucher <alexdeucher at gmail.com>
> Sent: Tuesday, April 29, 2025 3:05 PM
> To: Liu, Shaoyun <Shaoyun.Liu at amd.com>
> Cc: Chen, Michael <Michael.Chen at amd.com>; Deucher, Alexander <Alexander.Deucher at amd.com>; amd-gfx at lists.freedesktop.org; Khatri, Sunil <Sunil.Khatri at amd.com>; Liang, Prike <Prike.Liang at amd.com>
> Subject: Re: [PATCH 5/8] drm/amdgpu/mes12: add support for setting gang submit
>
> On Tue, Apr 29, 2025 at 2:56 PM Liu, Shaoyun <Shaoyun.Liu at amd.com> wrote:
> >
> > [AMD Official Use Only - AMD Internal Distribution Only]
> >
> > Correctly , the gang submission related stuff only used for scheduler pipe , kiq is for used for kernel driver only and  should not expose to the  user queue.
>
> What about kernel queues mapped through MES?  Do those need to go through KIQ or is SCHED ok?  They seem to go through KIQ today.  Is going through SCHED ok?
>
> Alex
>
>
> >
> > Regards
> > Shaoyun.liu
> >
> > -----Original Message-----
> > From: amd-gfx <amd-gfx-bounces at lists.freedesktop.org> On Behalf Of
> > Alex Deucher
> > Sent: Tuesday, April 29, 2025 2:39 PM
> > To: Chen, Michael <Michael.Chen at amd.com>
> > Cc: Deucher, Alexander <Alexander.Deucher at amd.com>;
> > amd-gfx at lists.freedesktop.org; Khatri, Sunil <Sunil.Khatri at amd.com>;
> > Liang, Prike <Prike.Liang at amd.com>
> > Subject: Re: [PATCH 5/8] drm/amdgpu/mes12: add support for setting
> > gang submit
> >
> > On Tue, Apr 29, 2025 at 2:23 PM Chen, Michael <Michael.Chen at amd.com> wrote:
> > >
> > > [Public]
> > >
> > >
> > >
> > >
> > > ________________________________
> > > From: amd-gfx <amd-gfx-bounces at lists.freedesktop.org> on behalf of
> > > Alex Deucher <alexander.deucher at amd.com>
> > > Sent: Monday, April 28, 2025 5:20 PM
> > > To: amd-gfx at lists.freedesktop.org <amd-gfx at lists.freedesktop.org>
> > > Cc: Deucher, Alexander <Alexander.Deucher at amd.com>; Khatri, Sunil
> > > <Sunil.Khatri at amd.com>; Liang, Prike <Prike.Liang at amd.com>
> > > Subject: [PATCH 5/8] drm/amdgpu/mes12: add support for setting gang
> > > submit
> > >
> > > Enable a primary and secondary queue that schedule together.
> > >
> > > v2: fix offset of api_status (Prike)
> > >
> > > Acked-by: Sunil Khatri <sunil.khatri at amd.com>
> > > Reviewed-by: Prike Liang <Prike.Liang at amd.com>
> > > Signed-off-by: Alex Deucher <alexander.deucher at amd.com>
> > > ---
> > >  drivers/gpu/drm/amd/amdgpu/mes_v12_0.c | 28
> > > ++++++++++++++++++++++++++
> > >  1 file changed, 28 insertions(+)
> > >
> > > diff --git a/drivers/gpu/drm/amd/amdgpu/mes_v12_0.c
> > > b/drivers/gpu/drm/amd/amdgpu/mes_v12_0.c
> > > index f9f2fbc0a7166..57d8b78210f9f 100644
> > > --- a/drivers/gpu/drm/amd/amdgpu/mes_v12_0.c
> > > +++ b/drivers/gpu/drm/amd/amdgpu/mes_v12_0.c
> > > @@ -879,6 +879,33 @@ static int mes_v12_0_reset_hw_queue(struct amdgpu_mes *mes,
> > >                          offsetof(union MESAPI__RESET, api_status));
> > > }
> > >
> > > +static int mes_v12_0_set_gang_submit(struct amdgpu_mes *mes,
> > > +                                    struct
> > > +mes_set_gang_submit_input
> > > +*input) {
> > > +       union MESAPI__SET_GANG_SUBMIT mes_gang_submit_pkt;
> > > +       int pipe;
> > > +
> > > +       memset(&mes_gang_submit_pkt, 0,
> > > + sizeof(mes_gang_submit_pkt));
> > > +
> > > +       mes_gang_submit_pkt.header.type = MES_API_TYPE_SCHEDULER;
> > > +       mes_gang_submit_pkt.header.opcode = MES_SCH_API_SET_GANG_SUBMIT;
> > > +       mes_gang_submit_pkt.header.dwsize =
> > > + API_FRAME_SIZE_IN_DWORDS;
> > > +
> > > +       mes_gang_submit_pkt.set_gang_submit.gang_context_addr =
> > > +               input->primary_gang_context_addr;
> > > +       mes_gang_submit_pkt.set_gang_submit.slave_gang_context_addr =
> > > +               input->secondary_gang_context_addr;
> > > +
> > > +       if (mes->adev->enable_uni_mes)
> > > +               pipe = AMDGPU_MES_KIQ_PIPE;
> > > +       else
> > > +               pipe = AMDGPU_MES_SCHED_PIPE;
> > >
> > > I think this packet should be always submitted to
> > > AMDGPU_MES_SCHED_PIPE, same pipe as in mes_v12_0_add_hw_queue where the queue is created.
> > > This is because HWS on AMDGPU_MES_KIQ_PIPE does not have the queue information.
> >
> > Sounds like the reset queue function and the map and unmap legacy queues should also be changed?
> >
> > Alex
> >
> > >
> > > +
> > > +       return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe,
> > > +                       &mes_gang_submit_pkt, sizeof(mes_gang_submit_pkt),
> > > +                       offsetof(union MESAPI__SET_GANG_SUBMIT,
> > > +api_status)); }
> > > +
> > >  static const struct amdgpu_mes_funcs mes_v12_0_funcs = {
> > >          .add_hw_queue = mes_v12_0_add_hw_queue,
> > >          .remove_hw_queue = mes_v12_0_remove_hw_queue, @@ -888,6
> > > +915,7 @@ static const struct amdgpu_mes_funcs mes_v12_0_funcs = {
> > >          .resume_gang = mes_v12_0_resume_gang,
> > >          .misc_op = mes_v12_0_misc_op,
> > >          .reset_hw_queue = mes_v12_0_reset_hw_queue,
> > > +       .set_gang_submit = mes_v12_0_set_gang_submit,
> > >  };
> > >
> > >  static int mes_v12_0_allocate_ucode_buffer(struct amdgpu_device
> > > *adev,
> > > --
> > > 2.49.0
> > >