[PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout

Joel Fernandes joelagnelf at nvidia.com
Sat May 3 04:07:56 UTC 2025


Add explanation of fwsec with diagrams. This helps clarify how the
nova-core falcon boot works.

Signed-off-by: Joel Fernandes <joelagnelf at nvidia.com>
---
 Documentation/gpu/nova/core/fwsec.rst | 180 ++++++++++++++++++++++++++
 Documentation/gpu/nova/index.rst      |   1 +
 2 files changed, 181 insertions(+)
 create mode 100644 Documentation/gpu/nova/core/fwsec.rst

diff --git a/Documentation/gpu/nova/core/fwsec.rst b/Documentation/gpu/nova/core/fwsec.rst
new file mode 100644
index 000000000000..bed941ac3f2b
--- /dev/null
+++ b/Documentation/gpu/nova/core/fwsec.rst
@@ -0,0 +1,180 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+=========================
+FWSEC (Firmware Security)
+=========================
+This document briefly/conceptually describes the FWSEC (Firmware Security) image
+and its role in the GPU boot sequence. As such, this information is subject to
+change in the future and is only current as of the Ampere GPU family. However,
+hopefully the concepts described will be useful for understanding the kernel code
+that deals with it. All the information is derived from publicly available
+sources such as public drivers and documentation.
+
+The role of FWSEC to provide secure boot, it is running in Heavy-secure mode. It does
+firmware verification after GPU reset and load various ucode images on to the other
+microcontrollers on the GPU such as the PMU and GSP.
+
+FWSEC itself is an application stored in the VBIOS ROM in the FWSEC partition of
+ROM (see vbios.rst for more details). It contains different commands like FRTS
+(Firmware Runtime Services) and SB (Secure Booting other microcontrollers after
+reset and loading them with other non-FWSEC ucode). The kernel driver only needs to
+to do FRTS, since SB is already already after reset by the time the kernel driver
+is loaded.
+
+The FRTS command carves out the WPR2 region (Write protected region) which contains
+data data required for power management. Once setup, only HS mode ucode can
+access it (see falcon.rst for privilege levels).
+
+The FWSEC image is located in the VBIOS ROM in the partition of the ROM that contains
+various ucode images (also known as applications) -- one of them being FWSEC. For how
+it is extracted, see vbios.rst and the vbios.rs source code.
+
+The Falcon data for each ucode images (including the FWSEC image) is a combination
+of headers, data sections (DMEM) and instruction code sections (IMEM). All these
+ucode images are stored in the same ROM partition and the PMU table is used to look
+up the application to load it based on its application ID (see vbios.rs).
+
+For the purposes of nova-core driver, the FWSEC contains an 'application interface'
+called DMEMMAPPER which is used to the "FWSEC-FRTS" command (among other commands it
+is capable of executing). For Ampere, FWSEC is running on the GSP in Heavy-secure
+mode and runs FRTS.
+
+FWSEC Memory Layout
+-------------------
+The memory layout of the FWSEC image is as follows (this is using an GA-102
+Ampere GPU as an example and could vary for future GPUs and is subject to change
+completely, it is just provided as an example):
+
+Here is a block diagram of the FWSEC memory layout::
+ ┌───────────────────────────────────────────────────────────────┐
+ │                         FWSEC ROM image (type 0xE0)           │
+ │                                                               │
+ │  ┌─────────────────────────────────┐                          │
+ │  │     PMU Falcon Ucode Table      │                          │
+ │  │     (PmuLookupTable)            │                          │
+ │  │  ┌─────────────────────────┐    │                          │
+ │  │  │ Table Header            │    │                          │
+ │  │  │ - version: 0x01         │    │                          │
+ │  │  │ - header_size: 6        │    │                          │
+ │  │  │ - entry_size: 6         │    │                          │
+ │  │  │ - entry_count: N        │    │                          │
+ │  │  │ - desc_version:3(unused)│    │                          │
+ │  │  └─────────────────────────┘    │                          │
+ │  │         ...                     │                          │
+ │  │  ┌─────────────────────────┐    │                          │
+ │  │  │ Entry for FWSEC (0x85)  │    │                          │
+ │  │  │ (PmuLookupTableEntry)   │    │                          │
+ │  │  │ - app_id: 0x85 (FWSEC)  │ ───┼────┐                     │
+ │  │  │ - target_id: 0x01 (PMU) │    │    │                     │
+ │  │  │ - data: offset ─────────┼────┼────┼───┐ look up FWSEC   │
+ │  │  └─────────────────────────┘    │    │   │ application.    │
+ │  └─────────────────────────────────┘    │   │                 │
+ │                                         │   │                 │
+ │                                         │   │                 │
+ │  ┌─────────────────────────────────┐    │   │                 │
+ │  │     FWSEC Ucode Component       │<───┘   │                 │
+ │  │     (aka Falcon data)           │        │                 │
+ │  │  ┌─────────────────────────┐    │        │                 │
+ │  │  │ FalconUCodeDescV3       │<───┼────────┘                 │
+ │  │  │ - hdr                   │    │                          │
+ │  │  │ - stored_size           │    │                          │
+ │  │  │ - pkc_data_offset       │    │                          │
+ │  │  │ - interface_offset ─────┼────┼────────────────┐         │
+ │  │  │ - imem_phys_base        │    │                │         │
+ │  │  │ - imem_load_size        │    │                │         │
+ │  │  │ - imem_virt_base        │    │                │         │
+ │  │  │ - dmem_phys_base        │    │                │         │
+ │  │  │ - dmem_load_size        │    │                │         │
+ │  │  │ - engine_id_mask        │    │                │         │
+ │  │  │ - ucode_id              │    │                │         │
+ │  │  │ - signature_count       │    │    look up sig │         │
+ │  │  │ - signature_versions --------------+          │         │
+ │  │  └─────────────────────────┘    │     │          │         │
+ │  │         (no gap)                │     │          │         │
+ │  │  ┌─────────────────────────┐    │     │          │         │
+ │  │  │ Signatures Section      │<───┼─────┘          │         │
+ │  │  │ (384 bytes per sig)     │    │                │         │
+ │  │  │ - RSA-3K Signature 1    │    │                │         │
+ │  │  │ - RSA-3K Signature 2    │    │                │         │
+ │  │  │   ...                   │    │                │         │
+ │  │  └─────────────────────────┘    │                │         │
+ │  │                                 │                │         │
+ │  │  ┌─────────────────────────┐    │                │         │
+ │  │  │ IMEM Section (Code)     │    │                │         │
+ │  │  │                         │    │                │         │
+ │  │  │ Contains instruction    │    │                │         │
+ │  │  │ code etc.               │    │                │         │
+ │  │  └─────────────────────────┘    │                │         │
+ │  │                                 │                │         │
+ │  │  ┌─────────────────────────┐    │                │         │
+ │  │  │ DMEM Section (Data)     │    │                │         │
+ │  │  │                         │    │                │         │
+ │  │  │ ┌─────────────────────┐ │    │                │         │
+ │  │  │ │ Application         │ │<───┼────────────────┘         │
+ │  │  │ │ Interface Table     │ │    │                          │
+ │  │  │ │ (FalconAppifHdrV1)  │ │    │                          │
+ │  │  │ │ Header:             │ │    │                          │
+ │  │  │ │ - version: 0x01     │ │    │                          │
+ │  │  │ │ - header_size: 4    │ │    │                          │
+ │  │  │ │ - entry_size: 8     │ │    │                          │
+ │  │  │ │ - entry_count: N    │ │    │                          │
+ │  │  │ │                     │ │    │                          │
+ │  │  │ │ Entries:            │ │    │                          │
+ │  │  │ │ ┌─────────────────┐ │ │    │                          │
+ │  │  │ │ │ DEVINIT (ID 1)  │ │ │    │                          │
+ │  │  │ │ │ - id: 0x01      │ │ │    │                          │
+ │  │  │ │ │ - dmemOffset X ─┼─┼─┼────┐                          │
+ │  │  │ │ └─────────────────┘ │ │    │                          │
+ │  │  │ │ ┌─────────────────┐ │ │    │                          │
+ │  │  │ │ │ DMEMMAPPER(ID 4)│ │ │    │                          │
+ │  │  │ │ │ - id: 0x04      │ │ │    │ Used only for DevInit    │
+ │  │  │ │ │  (NVFW_FALCON_  │ │ │    │ application (not FWSEC)  │
+ │  │  │ │ │   APPIF_ID_DMEMMAPPER)   │                          │
+ │  │  │ │ │ - dmemOffset Y ─┼─┼─┼────┼─────┐                    │
+ │  │  │ │ └─────────────────┘ │ │    │     │                    │
+ │  │  │ └─────────────────────┘ │    │     │                    │
+ │  │  │                         │    │     │                    │
+ │  │  │ ┌─────────────────────┐ │    │     │                    │
+ │  │  │ │ DEVINIT Engine      │<┼────┘     │ Used by FWSEC      │
+ │  │  │ │ Interface           │ │    │     │         app.       │
+ │  │  │ └─────────────────────┘ │    │     │                    │
+ │  │  │                         │    │     │                    │
+ │  │  │ ┌─────────────────────┐ │    │     │                    │
+ │  │  │ │ DMEM Mapper (ID 4)  │<┼────+─────┘                    │
+ │  │  │ │ (FalconAppifDmemmapperV3)  │                          │
+ │  │  │ │ - signature: "DMAP" │ │    │                          │
+ │  │  │ │ - version: 0x0003   │ │    │                          │
+ │  │  │ │ - Size: 64 bytes    │ │    │                          │
+ │  │  │ │ - cmd_in_buffer_off │ │────┼────────────┐             │
+ │  │  │ │ - cmd_in_buffer_size│ │    │            │             │
+ │  │  │ │ - cmd_out_buffer_off│ │────┼────────────┼─────┐       │
+ │  │  │ │ - cmd_out_buffer_sz │ │    │            │     │       │
+ │  │  │ │ - init_cmd          │ │    │            │     │       │
+ │  │  │ │ - features          │ │    │            │     │       │
+ │  │  │ │ - cmd_mask0/1       │ │    │            │     │       │
+ │  │  │ └─────────────────────┘ │    │            │     │       │
+ │  │  │                         │    │            │     │       │
+ │  │  │ ┌─────────────────────┐ │    │            │     │       │
+ │  │  │ │ Command Input Buffer│<┼────┼────────────┘     │       │
+ │  │  │ │ - Command data      │ │    │                  │       │
+ │  │  │ │ - Arguments         │ │    │                  │       │
+ │  │  │ └─────────────────────┘ │    │                  │       │
+ │  │  │                         │    │                  │       │
+ │  │  │ ┌─────────────────────┐ │    │                  │       │
+ │  │  │ │ Command Output      │<┼────┼──────────────────┘       │
+ │  │  │ │ Buffer              │ │    │                          │
+ │  │  │ │ - Results           │ │    │                          │
+ │  │  │ │ - Status            │ │    │                          │
+ │  │  │ └─────────────────────┘ │    │                          │
+ │  │  └─────────────────────────┘    │                          │
+ │  └─────────────────────────────────┘                          │
+ │                                                               │
+ └───────────────────────────────────────────────────────────────┘
+
+.. note::
+   The FWSEC image also plays a role in memory scrubbing (ECC initialization) and VPR
+   (Video Protected Region) initialization as well. Before the nova-core driver is even
+   loaded, the FWSEC image is running on the GSP in heavy-secure mode. After the devinit
+   sequence completes, it does VRAM memory scrubbing (ECC initialization). On consumer
+   GPUs, it scrubs only part of memory and then initiates 'async scrubbing'. Before this
+   async scrubbing completes, the unscrubbed VRAM cannot be used for allocation (thus DRM
+   memory allocators need to wait for this scrubbing to complete).
\ No newline at end of file
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 91cc802ed94f..22e5712ac6b0 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -28,4 +28,5 @@ vGPU manager VFIO driver and the nova-drm driver.
 
    core/guidelines
    core/vbios
+   core/fwsec
    core/todo
-- 
2.43.0



More information about the dri-devel mailing list