[PATCH v3 00/10] Preemption support for A7XX

[Rob Clark]

On Fri, Sep 6, 2024 at 12:59 PM Akhil P Oommen <quic_akhilpo at quicinc.com> wrote:
>
> On Thu, Sep 05, 2024 at 04:51:18PM +0200, Antonino Maniscalco wrote:
> > This series implements preemption for A7XX targets, which allows the GPU to
> > switch to an higher priority ring when work is pushed to it, reducing latency
> > for high priority submissions.
> >
> > This series enables L1 preemption with skip_save_restore which requires
> > the following userspace patches to function:
> >
> > https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30544
> >
> > A flag is added to `msm_submitqueue_create` to only allow submissions
> > from compatible userspace to be preempted, therefore maintaining
> > compatibility.
> >
> > Preemption is currently only enabled by default on A750, it can be
> > enabled on other targets through the `enable_preemption` module
> > parameter. This is because more testing is required on other targets.
> >
> > For testing on other HW it is sufficient to set that parameter to a
> > value of 1, then using the branch of mesa linked above, `TU_DEBUG=hiprio`
> > allows to run any application as high priority therefore preempting
> > submissions from other applications.
> >
> > The `msm_gpu_preemption_trigger` and `msm_gpu_preemption_irq` traces
> > added in this series can be used to observe preemption's behavior as
> > well as measuring preemption latency.
> >
> > Some commits from this series are based on a previous series to enable
> > preemption on A6XX targets:
> >
> > https://lkml.kernel.org/1520489185-21828-1-git-send-email-smasetty@codeaurora.org
> >
> > Signed-off-by: Antonino Maniscalco <antomani103 at gmail.com>
>
> Antonino, can you please test this once with per-process pt disabled to
> ensure that is not broken? It is handy sometimes while debugging.
> We just need to remove "adreno-smmu" compatible string from gpu smmu
> node in DT.

fwiw, I'd be ok supporting preemption on devices that have per-process
pgtables.  (And maybe we should be tainting the kernel if per-process
pgtables are disabled on a6xx+)

BR,
-R

> -Akhil.
>
> > ---
> > Changes in v3:
> > - Added documentation about preemption
> > - Use quirks to determine which target supports preemption
> > - Add a module parameter to force disabling or enabling preemption
> > - Clear postamble when profiling
> > - Define A6XX_CP_CONTEXT_SWITCH_CNTL_LEVEL fields in a6xx.xml
> > - Make preemption records MAP_PRIV
> > - Removed user ctx record (NON_PRIV) and patch 2/9 as it's not needed
> >   anymore
> > - Link to v2: https://lore.kernel.org/r/20240830-preemption-a750-t-v2-0-86aeead2cd80@gmail.com
> >
> > Changes in v2:
> > - Added preept_record_size for X185 in PATCH 3/7
> > - Added patches to reset perf counters
> > - Dropped unused defines
> > - Dropped unused variable (fixes warning)
> > - Only enable preemption on a750
> > - Reject MSM_SUBMITQUEUE_ALLOW_PREEMPT for unsupported targets
> > - Added Akhil's Reviewed-By tags to patches 1/9,2/9,3/9
> > - Added Neil's Tested-By tags
> > - Added explanation for UAPI changes in commit message
> > - Link to v1: https://lore.kernel.org/r/20240815-preemption-a750-t-v1-0-7bda26c34037@gmail.com
> >
> > ---
> > Antonino Maniscalco (10):
> >       drm/msm: Fix bv_fence being used as bv_rptr
> >       drm/msm: Add a `preempt_record_size` field
> >       drm/msm: Add CONTEXT_SWITCH_CNTL bitfields
> >       drm/msm/A6xx: Implement preemption for A7XX targets
> >       drm/msm/A6xx: Sync relevant adreno_pm4.xml changes
> >       drm/msm/A6xx: Use posamble to reset counters on preemption
> >       drm/msm/A6xx: Add traces for preemption
> >       drm/msm/A6XX: Add a flag to allow preemption to submitqueue_create
> >       drm/msm/A6xx: Enable preemption for A750
> >       Documentation: document adreno preemption
> >
> >  Documentation/gpu/msm-preemption.rst               |  98 +++++
> >  drivers/gpu/drm/msm/Makefile                       |   1 +
> >  drivers/gpu/drm/msm/adreno/a6xx_catalog.c          |   7 +-
> >  drivers/gpu/drm/msm/adreno/a6xx_gpu.c              | 331 +++++++++++++++-
> >  drivers/gpu/drm/msm/adreno/a6xx_gpu.h              | 166 ++++++++
> >  drivers/gpu/drm/msm/adreno/a6xx_preempt.c          | 430 +++++++++++++++++++++
> >  drivers/gpu/drm/msm/adreno/adreno_gpu.h            |   9 +-
> >  drivers/gpu/drm/msm/msm_drv.c                      |   4 +
> >  drivers/gpu/drm/msm/msm_gpu_trace.h                |  28 ++
> >  drivers/gpu/drm/msm/msm_ringbuffer.h               |   8 +
> >  drivers/gpu/drm/msm/msm_submitqueue.c              |   3 +
> >  drivers/gpu/drm/msm/registers/adreno/a6xx.xml      |   7 +-
> >  .../gpu/drm/msm/registers/adreno/adreno_pm4.xml    |  39 +-
> >  include/uapi/drm/msm_drm.h                         |   5 +-
> >  14 files changed, 1094 insertions(+), 42 deletions(-)
> > ---
> > base-commit: 7c626ce4bae1ac14f60076d00eafe71af30450ba
> > change-id: 20240815-preemption-a750-t-fcee9a844b39
> >
> > Best regards,
> > --
> > Antonino Maniscalco <antomani103 at gmail.com>
> >