[PATCH v4 00/14] RFC Support hot device unplug in amdgpu
andrey.grodzovsky at amd.com
Mon Jan 18 21:01:09 UTC 2021
Until now extracting a card either by physical extraction (e.g. eGPU with
thunderbolt connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove)
would cause random crashes in user apps. The random crashes in apps were
mostly due to the app having mapped a device backed BO into its address
space was still trying to access the BO while the backing device was gone.
To answer this first problem Christian suggested to fix the handling of mapped
memory in the clients when the device goes away by forcibly unmap all buffers the
user processes has by clearing their respective VMAs mapping the device BOs.
Then when the VMAs try to fill in the page tables again we check in the fault
handlerif the device is removed and if so, return an error. This will generate a
SIGBUS to the application which can then cleanly terminate.This indeed was done
but this in turn created a problem of kernel OOPs were the OOPSes were due to the
fact that while the app was terminating because of the SIGBUSit would trigger use
after free in the driver by calling to accesses device structures that were already
released from the pci remove sequence.This was handled by introducing a 'flush'
sequence during device removal were we wait for drm file reference to drop to 0
meaning all user clients directly using this device terminated.
Based on discussions in the mailing list with Daniel and Pekka  and based on the document
produced by Pekka from those discussions  the whole approach with returning SIGBUS and
waiting for all user clients having CPU mapping of device BOs to die was dropped.
Instead as per the document suggestion the device structures are kept alive until
the last reference to the device is dropped by user client and in the meanwhile all existing and new CPU mappings of the BOs
belonging to the device directly or by dma-buf import are rerouted to per user
process dummy rw page.Also, I skipped the 'Requirements for KMS UAPI' section of 
since i am trying to get the minimal set of requirements that still give useful solution
to work and this is the'Requirements for Render and Cross-Device UAPI' section and so my
test case is removing a secondary device, which is render only and is not involved
More updates following comments from v2 such as removing loop to find DRM file when rerouting
page faults to dummy page,getting rid of unnecessary sysfs handling refactoring and moving
prevention of GPU recovery post device unplug from amdgpu to scheduler layer.
On top of that added unplug support for the IOMMU enabled system.
Drop last sysfs hack and use sysfs default attribute.
Guard against write accesses after device removal to avoid modifying released memory.
Update dummy pages handling to on demand allocation and release through drm managed framework.
Add return value to scheduler job TO handler (by Luben Tuikov) and use this in amdgpu for prevention
of GPU recovery post device unplug
Also rebase on top of drm-misc-mext instead of amd-staging-drm-next
With these patches I am able to gracefully remove the secondary card using sysfs remove hook while glxgears
is running off of secondary card (DRI_PRIME=1) without kernel oopses or hangs and keep working
with the primary card or soft reset the device without hangs or oopses
TODOs for followup work:
Convert AMDGPU code to use devm (for hw stuff) and drmm (for sw stuff and allocations) (Daniel)
Support plugging the secondary device back after unplug - currently still experiencing HW error on plugging back.
Add support for 'Requirements for KMS UAPI' section of  - unplugging primary, display connected card.
 - Discussions during v3 of the patchset https://www.spinics.net/lists/amd-gfx/msg55576.html
 - drm/doc: device hot-unplug for userspace https://www.spinics.net/lists/dri-devel/msg259755.html
 - Related gitlab ticket https://gitlab.freedesktop.org/drm/amd/-/issues/1081
Andrey Grodzovsky (13):
drm/ttm: Remap all page faults to per process dummy page.
drm: Unamp the entire device address space on device unplug
drm/ttm: Expose ttm_tt_unpopulate for driver use
drm/sched: Cancel and flush all oustatdning jobs before finish.
drm/amdgpu: Split amdgpu_device_fini into early and late
drm/amdgpu: Add early fini callback
drm/amdgpu: Register IOMMU topology notifier per device.
drm/amdgpu: Fix a bunch of sdma code crash post device unplug
drm/amdgpu: Remap all page faults to per process dummy page.
dmr/amdgpu: Move some sysfs attrs creation to default_attr
drm/amdgpu: Guard against write accesses after device removal
drm/sched: Make timeout timer rearm conditional.
drm/amdgpu: Prevent any job recoveries after device is unplugged.
Luben Tuikov (1):
drm/scheduler: Job timeout handler returns status
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 11 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c | 17 +--
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 149 ++++++++++++++++++++--
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 20 ++-
drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c | 15 ++-
drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_gart.h | 1 +
drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 9 ++
drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c | 25 ++--
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 26 ++--
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h | 3 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 19 ++-
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 12 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 10 ++
drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 2 +
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 53 +++++---
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h | 3 +
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c | 1 +
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 70 ++++++++++
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h | 52 +-------
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 21 ++-
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 8 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 14 +-
drivers/gpu/drm/amd/amdgpu/cik_ih.c | 2 +-
drivers/gpu/drm/amd/amdgpu/cz_ih.c | 2 +-
drivers/gpu/drm/amd/amdgpu/iceland_ih.c | 2 +-
drivers/gpu/drm/amd/amdgpu/navi10_ih.c | 2 +-
drivers/gpu/drm/amd/amdgpu/psp_v11_0.c | 16 +--
drivers/gpu/drm/amd/amdgpu/psp_v12_0.c | 8 +-
drivers/gpu/drm/amd/amdgpu/psp_v3_1.c | 8 +-
drivers/gpu/drm/amd/amdgpu/si_ih.c | 2 +-
drivers/gpu/drm/amd/amdgpu/tonga_ih.c | 2 +-
drivers/gpu/drm/amd/amdgpu/vega10_ih.c | 2 +-
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c | 12 +-
drivers/gpu/drm/amd/include/amd_shared.h | 2 +
drivers/gpu/drm/drm_drv.c | 3 +
drivers/gpu/drm/etnaviv/etnaviv_sched.c | 10 +-
drivers/gpu/drm/lima/lima_sched.c | 4 +-
drivers/gpu/drm/panfrost/panfrost_job.c | 9 +-
drivers/gpu/drm/scheduler/sched_main.c | 18 ++-
drivers/gpu/drm/ttm/ttm_bo_vm.c | 82 +++++++++++-
drivers/gpu/drm/ttm/ttm_tt.c | 1 +
drivers/gpu/drm/v3d/v3d_sched.c | 32 ++---
include/drm/gpu_scheduler.h | 17 ++-
include/drm/ttm/ttm_bo_api.h | 2 +
45 files changed, 583 insertions(+), 198 deletions(-)
More information about the amd-gfx