[RFC PATCH 0/8] ULLS for kernel submission of migration jobs

[Thomas Hellström]

Hi, Matt,

On Sun, 2024-08-11 at 19:47 -0700, Matthew Brost wrote:
> Ultra low latency for kernel submission of migration jobs.
> 
> The basic idea is that faults (CPU or GPU) typically depend on
> migration
> jobs. Faults should be addressed as quickly as possible, but context
> switches via GuC on hardware are slow. To avoid context switches,
> perform ULLS in the kernel for migration jobs on discrete faulting
> devices with an LR VM open.
> 
> This is implemented by switching the migration layer to ULLS mode
> upon
> opening an LR VM. In ULLS mode, migration jobs have a preamble and
> postamble: the preamble clears the current semaphore value, and the
> postamble waits for the next semaphore value. Each job submission
> sets
> the current semaphore in memory, bypassing the GuC. The net effect is
> that the migration execution queue never gets switched off the
> hardware
> while an LR VM is open.
> 
> There may be concerns regarding power management, as the ring program
> continuously runs on a copy engine, and a force wake reference to a
> copy
> engine is held with an LR VM open.
> 
> The implementation has been lightly tested but seems to be working.
> 
> This approach will likely be put on hold until SVM is operational
> with
> benchmarks, but it is being posted early for feedback and as a public
> checkpoint.
> 
> Matt

The main concern I have with this is that, at least according to
upstream discussions, pagefaults are so slow anyway, a performant stack
needs to try extremely hard to avoid them using manual prefaults, and
if we hit a gpu pagefault, we've lost anyway and any migration latency
optimization won't matter much.

Also, for power management, LR VM open is a very simple strategy, which
is good, but shouldn't it be possible to hook that up to LR job
running, similar to vm->preempt.rebind_deactivated?

/Thomas


> 
> Matthew Brost (8):
>   drm/xe: Add xe_hw_engine_write_ring_tail
>   drm/xe: Add ULLS support to LRC
>   drm/xe: Add ULLS flags for jobs
>   drm/xe: Add ULLS migration job support to migration layer
>   drm/xe: Add MI_SEMAPHORE_WAIT instruction defs
>   drm/xe: Add ULLS migration job support to ring ops
>   drm/xe: Add ULLS migration job support to GuC submission
>   drm/xe: Enable ULLS migration jobs when opening LR VM
> 
>  .../gpu/drm/xe/instructions/xe_mi_commands.h  |   6 +
>  drivers/gpu/drm/xe/xe_guc_submit.c            |  26 +++-
>  drivers/gpu/drm/xe/xe_hw_engine.c             |  10 ++
>  drivers/gpu/drm/xe/xe_hw_engine.h             |   1 +
>  drivers/gpu/drm/xe/xe_lrc.c                   |  49 +++++++
>  drivers/gpu/drm/xe/xe_lrc.h                   |   3 +
>  drivers/gpu/drm/xe/xe_lrc_types.h             |   2 +
>  drivers/gpu/drm/xe/xe_migrate.c               | 130
> +++++++++++++++++-
>  drivers/gpu/drm/xe/xe_migrate.h               |   4 +
>  drivers/gpu/drm/xe/xe_ring_ops.c              |  32 +++++
>  drivers/gpu/drm/xe/xe_sched_job_types.h       |   3 +
>  drivers/gpu/drm/xe/xe_vm.c                    |  10 ++
>  12 files changed, 268 insertions(+), 8 deletions(-)
>