[Intel-xe] [PATCH 1/3] drm/xe: Kill xe_bo_doc.h

[Rodrigo Vivi]

Keep the documentation inside the code.
No update to the documentation itself at this time.
Let's first ensure that the entire documentation is closer
to the relevant code, otherwise it will get very easily outdated.

Signed-off-by: Rodrigo Vivi <rodrigo.vivi at intel.com>
---
 Documentation/gpu/xe/xe_mm.rst |   2 +-
 drivers/gpu/drm/xe/xe_bo.c     | 170 +++++++++++++++++++++++++++++++
 drivers/gpu/drm/xe/xe_bo_doc.h | 179 ---------------------------------
 3 files changed, 171 insertions(+), 180 deletions(-)
 delete mode 100644 drivers/gpu/drm/xe/xe_bo_doc.h

diff --git a/Documentation/gpu/xe/xe_mm.rst b/Documentation/gpu/xe/xe_mm.rst
index 6c8fd8b4a466..bcb5ad56da9a 100644
--- a/Documentation/gpu/xe/xe_mm.rst
+++ b/Documentation/gpu/xe/xe_mm.rst
@@ -4,7 +4,7 @@
 Memory Management
 =================
 
-.. kernel-doc:: drivers/gpu/drm/xe/xe_bo_doc.h
+.. kernel-doc:: drivers/gpu/drm/xe/xe_bo.c
    :doc: Buffer Objects (BO)
 
 Pagetable building
diff --git a/drivers/gpu/drm/xe/xe_bo.c b/drivers/gpu/drm/xe/xe_bo.c
index 61789c0e88fb..8ced990e3982 100644
--- a/drivers/gpu/drm/xe/xe_bo.c
+++ b/drivers/gpu/drm/xe/xe_bo.c
@@ -27,6 +27,176 @@
 #include "xe_ttm_stolen_mgr.h"
 #include "xe_vm.h"
 
+/**
+ * DOC: Buffer Objects (BO)
+ *
+ * BO management
+ * =============
+ *
+ * TTM manages (placement, eviction, etc...) all BOs in XE.
+ *
+ * BO creation
+ * ===========
+ *
+ * Create a chunk of memory which can be used by the GPU. Placement rules
+ * (sysmem or vram region) passed in upon creation. TTM handles placement of BO
+ * and can trigger eviction of other BOs to make space for the new BO.
+ *
+ * Kernel BOs
+ * ----------
+ *
+ * A kernel BO is created as part of driver load (e.g. uC firmware images, GuC
+ * ADS, etc...) or a BO created as part of a user operation which requires
+ * a kernel BO (e.g. engine state, memory for page tables, etc...). These BOs
+ * are typically mapped in the GGTT (any kernel BOs aside memory for page tables
+ * are in the GGTT), are pinned (can't move or be evicted at runtime), have a
+ * vmap (XE can access the memory via xe_map layer) and have contiguous physical
+ * memory.
+ *
+ * More details of why kernel BOs are pinned and contiguous below.
+ *
+ * User BOs
+ * --------
+ *
+ * A user BO is created via the DRM_IOCTL_XE_GEM_CREATE IOCTL. Once it is
+ * created the BO can be mmap'd (via DRM_IOCTL_XE_GEM_MMAP_OFFSET) for user
+ * access and it can be bound for GPU access (via DRM_IOCTL_XE_VM_BIND). All
+ * user BOs are evictable and user BOs are never pinned by XE. The allocation of
+ * the backing store can be defered from creation time until first use which is
+ * either mmap, bind, or pagefault.
+ *
+ * Private BOs
+ * ~~~~~~~~~~~
+ *
+ * A private BO is a user BO created with a valid VM argument passed into the
+ * create IOCTL. If a BO is private it cannot be exported via prime FD and
+ * mappings can only be created for the BO within the VM it is tied to. Lastly,
+ * the BO dma-resv slots / lock point to the VM's dma-resv slots / lock (all
+ * private BOs to a VM share common dma-resv slots / lock).
+ *
+ * External BOs
+ * ~~~~~~~~~~~~
+ *
+ * An external BO is a user BO created with a NULL VM argument passed into the
+ * create IOCTL. An external BO can be shared with different UMDs / devices via
+ * prime FD and the BO can be mapped into multiple VMs. An external BO has its
+ * own unique dma-resv slots / lock. An external BO will be in an array of all
+ * VMs which has a mapping of the BO. This allows VMs to lookup and lock all
+ * external BOs mapped in the VM as needed.
+ *
+ * BO placement
+ * ~~~~~~~~~~~~
+ *
+ * When a user BO is created, a mask of valid placements is passed indicating
+ * which memory regions are considered valid.
+ *
+ * The memory region information is available via query uAPI (TODO: add link).
+ *
+ * BO validation
+ * =============
+ *
+ * BO validation (ttm_bo_validate) refers to ensuring a BO has a valid
+ * placement. If a BO was swapped to temporary storage, a validation call will
+ * trigger a move back to a valid (location where GPU can access BO) placement.
+ * Validation of a BO may evict other BOs to make room for the BO being
+ * validated.
+ *
+ * BO eviction / moving
+ * ====================
+ *
+ * All eviction (or in other words, moving a BO from one memory location to
+ * another) is routed through TTM with a callback into XE.
+ *
+ * Runtime eviction
+ * ----------------
+ *
+ * Runtime evictions refers to during normal operations where TTM decides it
+ * needs to move a BO. Typically this is because TTM needs to make room for
+ * another BO and the evicted BO is first BO on LRU list that is not locked.
+ *
+ * An example of this is a new BO which can only be placed in VRAM but there is
+ * not space in VRAM. There could be multiple BOs which have sysmem and VRAM
+ * placement rules which currently reside in VRAM, TTM trigger a will move of
+ * one (or multiple) of these BO(s) until there is room in VRAM to place the new
+ * BO. The evicted BO(s) are valid but still need new bindings before the BO
+ * used again (exec or compute mode rebind worker).
+ *
+ * Another example would be, TTM can't find a BO to evict which has another
+ * valid placement. In this case TTM will evict one (or multiple) unlocked BO(s)
+ * to a temporary unreachable (invalid) placement. The evicted BO(s) are invalid
+ * and before next use need to be moved to a valid placement and rebound.
+ *
+ * In both cases, moves of these BOs are scheduled behind the fences in the BO's
+ * dma-resv slots.
+ *
+ * WW locking tries to ensures if 2 VMs use 51% of the memory forward progress
+ * is made on both VMs.
+ *
+ * Runtime eviction uses per a GT migration engine (TODO: link to migration
+ * engine doc) to do a GPU memcpy from one location to another.
+ *
+ * Rebinds after runtime eviction
+ * ------------------------------
+ *
+ * When BOs are moved, every mapping (VMA) of the BO needs to rebound before
+ * the BO is used again. Every VMA is added to an evicted list of its VM when
+ * the BO is moved. This is safe because of the VM locking structure (TODO: link
+ * to VM locking doc). On the next use of a VM (exec or compute mode rebind
+ * worker) the evicted VMA list is checked and rebinds are triggered. In the
+ * case of faulting VM, the rebind is done in the page fault handler.
+ *
+ * Suspend / resume eviction of VRAM
+ * ---------------------------------
+ *
+ * During device suspend / resume VRAM may lose power which means the contents
+ * of VRAM's memory is blown away. Thus BOs present in VRAM at the time of
+ * suspend must be moved to sysmem in order for their contents to be saved.
+ *
+ * A simple TTM call (ttm_resource_manager_evict_all) can move all non-pinned
+ * (user) BOs to sysmem. External BOs that are pinned need to be manually
+ * evicted with a simple loop + xe_bo_evict call. It gets a little trickier
+ * with kernel BOs.
+ *
+ * Some kernel BOs are used by the GT migration engine to do moves, thus we
+ * can't move all of the BOs via the GT migration engine. For simplity, use a
+ * TTM memcpy (CPU) to move any kernel (pinned) BO on either suspend or resume.
+ *
+ * Some kernel BOs need to be restored to the exact same physical location. TTM
+ * makes this rather easy but the caveat is the memory must be contiguous. Again
+ * for simplity, we enforce that all kernel (pinned) BOs are contiguous and
+ * restored to the same physical location.
+ *
+ * Pinned external BOs in VRAM are restored on resume via the GPU.
+ *
+ * Rebinds after suspend / resume
+ * ------------------------------
+ *
+ * Most kernel BOs have GGTT mappings which must be restored during the resume
+ * process. All user BOs are rebound after validation on their next use.
+ *
+ * Future work
+ * ===========
+ *
+ * Trim the list of BOs which is saved / restored via TTM memcpy on suspend /
+ * resume. All we really need to save / restore via TTM memcpy is the memory
+ * required for the GuC to load and the memory for the GT migrate engine to
+ * operate.
+ *
+ * Do not require kernel BOs to be contiguous in physical memory / restored to
+ * the same physical address on resume. In all likelihood the only memory that
+ * needs to be restored to the same physical address is memory used for page
+ * tables. All of that memory is allocated 1 page at time so the contiguous
+ * requirement isn't needed. Some work on the vmap code would need to be done if
+ * kernel BOs are not contiguous too.
+ *
+ * Make some kernel BO evictable rather than pinned. An example of this would be
+ * engine state, in all likelihood if the dma-slots of these BOs where properly
+ * used rather than pinning we could safely evict + rebind these BOs as needed.
+ *
+ * Some kernel BOs do not need to be restored on resume (e.g. GuC ADS as that is
+ * repopulated on resume), add flag to mark such objects as no save / restore.
+ */
+
 static const struct ttm_place sys_placement_flags = {
 	.fpfn = 0,
 	.lpfn = 0,
diff --git a/drivers/gpu/drm/xe/xe_bo_doc.h b/drivers/gpu/drm/xe/xe_bo_doc.h
deleted file mode 100644
index f57d440cc95a..000000000000
--- a/drivers/gpu/drm/xe/xe_bo_doc.h
+++ /dev/null
@@ -1,179 +0,0 @@
-/* SPDX-License-Identifier: MIT */
-/*
- * Copyright © 2022 Intel Corporation
- */
-
-#ifndef _XE_BO_DOC_H_
-#define _XE_BO_DOC_H_
-
-/**
- * DOC: Buffer Objects (BO)
- *
- * BO management
- * =============
- *
- * TTM manages (placement, eviction, etc...) all BOs in XE.
- *
- * BO creation
- * ===========
- *
- * Create a chunk of memory which can be used by the GPU. Placement rules
- * (sysmem or vram region) passed in upon creation. TTM handles placement of BO
- * and can trigger eviction of other BOs to make space for the new BO.
- *
- * Kernel BOs
- * ----------
- *
- * A kernel BO is created as part of driver load (e.g. uC firmware images, GuC
- * ADS, etc...) or a BO created as part of a user operation which requires
- * a kernel BO (e.g. engine state, memory for page tables, etc...). These BOs
- * are typically mapped in the GGTT (any kernel BOs aside memory for page tables
- * are in the GGTT), are pinned (can't move or be evicted at runtime), have a
- * vmap (XE can access the memory via xe_map layer) and have contiguous physical
- * memory.
- *
- * More details of why kernel BOs are pinned and contiguous below.
- *
- * User BOs
- * --------
- *
- * A user BO is created via the DRM_IOCTL_XE_GEM_CREATE IOCTL. Once it is
- * created the BO can be mmap'd (via DRM_IOCTL_XE_GEM_MMAP_OFFSET) for user
- * access and it can be bound for GPU access (via DRM_IOCTL_XE_VM_BIND). All
- * user BOs are evictable and user BOs are never pinned by XE. The allocation of
- * the backing store can be defered from creation time until first use which is
- * either mmap, bind, or pagefault.
- *
- * Private BOs
- * ~~~~~~~~~~~
- *
- * A private BO is a user BO created with a valid VM argument passed into the
- * create IOCTL. If a BO is private it cannot be exported via prime FD and
- * mappings can only be created for the BO within the VM it is tied to. Lastly,
- * the BO dma-resv slots / lock point to the VM's dma-resv slots / lock (all
- * private BOs to a VM share common dma-resv slots / lock).
- *
- * External BOs
- * ~~~~~~~~~~~~
- *
- * An external BO is a user BO created with a NULL VM argument passed into the
- * create IOCTL. An external BO can be shared with different UMDs / devices via
- * prime FD and the BO can be mapped into multiple VMs. An external BO has its
- * own unique dma-resv slots / lock. An external BO will be in an array of all
- * VMs which has a mapping of the BO. This allows VMs to lookup and lock all
- * external BOs mapped in the VM as needed.
- *
- * BO placement
- * ~~~~~~~~~~~~
- *
- * When a user BO is created, a mask of valid placements is passed indicating
- * which memory regions are considered valid.
- *
- * The memory region information is available via query uAPI (TODO: add link).
- *
- * BO validation
- * =============
- *
- * BO validation (ttm_bo_validate) refers to ensuring a BO has a valid
- * placement. If a BO was swapped to temporary storage, a validation call will
- * trigger a move back to a valid (location where GPU can access BO) placement.
- * Validation of a BO may evict other BOs to make room for the BO being
- * validated.
- *
- * BO eviction / moving
- * ====================
- *
- * All eviction (or in other words, moving a BO from one memory location to
- * another) is routed through TTM with a callback into XE.
- *
- * Runtime eviction
- * ----------------
- *
- * Runtime evictions refers to during normal operations where TTM decides it
- * needs to move a BO. Typically this is because TTM needs to make room for
- * another BO and the evicted BO is first BO on LRU list that is not locked.
- *
- * An example of this is a new BO which can only be placed in VRAM but there is
- * not space in VRAM. There could be multiple BOs which have sysmem and VRAM
- * placement rules which currently reside in VRAM, TTM trigger a will move of
- * one (or multiple) of these BO(s) until there is room in VRAM to place the new
- * BO. The evicted BO(s) are valid but still need new bindings before the BO
- * used again (exec or compute mode rebind worker).
- *
- * Another example would be, TTM can't find a BO to evict which has another
- * valid placement. In this case TTM will evict one (or multiple) unlocked BO(s)
- * to a temporary unreachable (invalid) placement. The evicted BO(s) are invalid
- * and before next use need to be moved to a valid placement and rebound.
- *
- * In both cases, moves of these BOs are scheduled behind the fences in the BO's
- * dma-resv slots.
- *
- * WW locking tries to ensures if 2 VMs use 51% of the memory forward progress
- * is made on both VMs.
- *
- * Runtime eviction uses per a GT migration engine (TODO: link to migration
- * engine doc) to do a GPU memcpy from one location to another.
- *
- * Rebinds after runtime eviction
- * ------------------------------
- *
- * When BOs are moved, every mapping (VMA) of the BO needs to rebound before
- * the BO is used again. Every VMA is added to an evicted list of its VM when
- * the BO is moved. This is safe because of the VM locking structure (TODO: link
- * to VM locking doc). On the next use of a VM (exec or compute mode rebind
- * worker) the evicted VMA list is checked and rebinds are triggered. In the
- * case of faulting VM, the rebind is done in the page fault handler.
- *
- * Suspend / resume eviction of VRAM
- * ---------------------------------
- *
- * During device suspend / resume VRAM may lose power which means the contents
- * of VRAM's memory is blown away. Thus BOs present in VRAM at the time of
- * suspend must be moved to sysmem in order for their contents to be saved.
- *
- * A simple TTM call (ttm_resource_manager_evict_all) can move all non-pinned
- * (user) BOs to sysmem. External BOs that are pinned need to be manually
- * evicted with a simple loop + xe_bo_evict call. It gets a little trickier
- * with kernel BOs.
- *
- * Some kernel BOs are used by the GT migration engine to do moves, thus we
- * can't move all of the BOs via the GT migration engine. For simplity, use a
- * TTM memcpy (CPU) to move any kernel (pinned) BO on either suspend or resume.
- *
- * Some kernel BOs need to be restored to the exact same physical location. TTM
- * makes this rather easy but the caveat is the memory must be contiguous. Again
- * for simplity, we enforce that all kernel (pinned) BOs are contiguous and
- * restored to the same physical location.
- *
- * Pinned external BOs in VRAM are restored on resume via the GPU.
- *
- * Rebinds after suspend / resume
- * ------------------------------
- *
- * Most kernel BOs have GGTT mappings which must be restored during the resume
- * process. All user BOs are rebound after validation on their next use.
- *
- * Future work
- * ===========
- *
- * Trim the list of BOs which is saved / restored via TTM memcpy on suspend /
- * resume. All we really need to save / restore via TTM memcpy is the memory
- * required for the GuC to load and the memory for the GT migrate engine to
- * operate.
- *
- * Do not require kernel BOs to be contiguous in physical memory / restored to
- * the same physical address on resume. In all likelihood the only memory that
- * needs to be restored to the same physical address is memory used for page
- * tables. All of that memory is allocated 1 page at time so the contiguous
- * requirement isn't needed. Some work on the vmap code would need to be done if
- * kernel BOs are not contiguous too.
- *
- * Make some kernel BO evictable rather than pinned. An example of this would be
- * engine state, in all likelihood if the dma-slots of these BOs where properly
- * used rather than pinning we could safely evict + rebind these BOs as needed.
- *
- * Some kernel BOs do not need to be restored on resume (e.g. GuC ADS as that is
- * repopulated on resume), add flag to mark such objects as no save / restore.
- */
-
-#endif
-- 
2.41.0