[PATCH v2 13/13] media: uapi: mediatek: document ISP7x camera system and user controls
shangyao lin
shangyao.lin at mediatek.com
Mon Jul 7 01:31:54 UTC 2025
From: "shangyao.lin" <shangyao.lin at mediatek.com>
This document covers:
- The hardware pipeline and firmware interface
- Integration with the V4L2 CAMSYS/media controller framework
- The list of custom V4L2 controls, their usage, valid ranges, and typical usage flow
Signed-off-by: shangyao.lin <shangyao.lin at mediatek.com>
---
.../media/v4l/mtk-isp7x-camsys.rst | 94 +++++++++
.../media/v4l/mtk-isp7x-controls.rst | 199 ++++++++++++++++++
2 files changed, 293 insertions(+)
create mode 100755 Documentation/userspace-api/media/v4l/mtk-isp7x-camsys.rst
create mode 100755 Documentation/userspace-api/media/v4l/mtk-isp7x-controls.rst
diff --git a/Documentation/userspace-api/media/v4l/mtk-isp7x-camsys.rst b/Documentation/userspace-api/media/v4l/mtk-isp7x-camsys.rst
new file mode 100755
index 000000000000..04a1fe1c448d
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/mtk-isp7x-camsys.rst
@@ -0,0 +1,94 @@
+==========================
+MediaTek ISP7x Camera System
+==========================
+
+Overview
+========
+
+The MediaTek ISP7x camera system is designed to provide a flexible, high-performance image capture pipeline for SoCs, leveraging the Linux V4L2 framework. The system is composed of multiple hardware modules (Sensor, SENINF, RAW ISP, etc.) and a firmware backend, with a clear separation of control and data flow. This document describes the hardware architecture from the perspective of the interface exposed to the firmware (backend), and how it integrates with the V4L2 CAMSYS framework.
+
+V4L2 CAMSYS Framework Overview
+==============================
+
+The V4L2 CAMSYS framework models the camera pipeline as a set of interconnected devices and subdevices:
+
+- **Video Device**: Exposes buffer queue/dequeue (enque/deque) operations to user space, allowing applications to stream image data.
+- **Subdevice**: Represents a hardware block (e.g., sensor, SENINF, RAW ISP) and handles control/configuration operations.
+- **Pads and Links**: Each subdevice has pads (input/output endpoints). Media links connect pads, representing the physical and logical data flow between hardware blocks.
+
+The topology and link state are established at probe time and can be dynamically enabled/disabled by user space, depending on the desired pipeline configuration.
+
+Hardware Pipeline
+=================
+
+A typical pipeline consists of:
+
+.. code-block::
+
+ Sensor --> SENINF --> RAW ISP --> (optional: YUV ISP) --> Memory
+
+- **Sensor**: Provides MIPI CSI-2 data.
+- **SENINF**: Receives and demultiplexes sensor data, supports virtual channel (VC) and data type (DT) routing.
+- **RAW ISP**: Main image signal processor, handles demosaicing, noise reduction, and other image enhancements.
+- **YUV ISP** (optional): Further processing for YUV output.
+- **CAMSYS**: System controller, manages pipeline state, buffer flow, and synchronization.
+
+Firmware Interface
+==================
+
+The firmware (backend, e.g., CCD/SCP) is responsible for:
+
+- Calculating hardware settings (3A, tuning, DMA, FBC, Twin, etc.) for each frame.
+- Generating command queues (CQ) and descriptors to program the hardware.
+- Abstracting hardware-specific details from the kernel driver.
+
+The kernel driver communicates with the firmware via an inter-processor interface (IPI), typically using RPMSG. The main interactions are:
+
+1. **Session Management**: The kernel establishes a session with the firmware at stream-on, and destroys it at stream-off.
+2. **Per-frame Request Scheduling**: For each user request (frame), the kernel sends a request to the firmware, including sensor settings, buffer addresses, and control parameters.
+3. **CQ Generation**: The firmware generates the necessary command queues and returns them to the kernel.
+4. **Event Notification**: The firmware notifies the kernel of CQ completion, errors, or other events.
+
+Request Scheduling and Buffer Flow
+==================================
+
+The request scheduling flow is as follows:
+
+1. **User space** submits a request (with buffer and control settings) via V4L2.
+2. **PipeMgr** (user space) collects requests, groups them by file descriptor, and sends them to the kernel CAMSYS driver.
+3. **CAMSYS** schedules requests, manages buffer flow, and coordinates with the firmware for hardware programming.
+4. **Firmware** processes each request, generates CQ, and signals completion.
+5. **Kernel** dequeues the buffer and notifies user space.
+
+The system supports per-frame control, allowing sensor settings and ISP parameters to be updated on a frame-by-frame basis.
+
+Media Controller Topology
+=========================
+
+The media controller graph models the hardware pipeline as a set of entities (video devices, subdevices) connected by links. Each entity exposes pads, and links are established at probe time based on the hardware configuration and device tree.
+
+.. code-block::
+
+ +---------+ +--------+ +--------+ +------+
+ | Sensor | --> | SENINF | --> | RAW ISP| --> | Video|
+ +---------+ +--------+ +--------+ +------+
+
+- Solid lines: active links (enabled)
+- Dashed lines: inactive links (disabled)
+
+User space can query and configure the topology using standard V4L2 and media controller APIs.
+
+Backend Abstraction
+===================
+
+The backend firmware abstracts hardware-specific programming, allowing the kernel driver to remain generic and maintainable. All hardware register programming, tuning, and advanced features are handled in the firmware, with the kernel acting as a control and data flow manager.
+
+This separation ensures that proprietary hardware details remain in the firmware, while the kernel driver exposes a standard V4L2 interface.
+
+Summary
+=======
+
+- The MediaTek ISP7x camera system is built on the V4L2 media controller framework, with a modular pipeline of video devices and subdevices.
+- The kernel driver manages pipeline state, buffer flow, and per-frame control, while delegating hardware programming to the backend firmware via IPI.
+- The firmware interface is session-based, supports per-frame CQ generation, and abstracts all hardware-specific details from the kernel.
+- The media controller graph models the hardware pipeline, allowing flexible configuration and dynamic topology management.
\ No newline at end of file
diff --git a/Documentation/userspace-api/media/v4l/mtk-isp7x-controls.rst b/Documentation/userspace-api/media/v4l/mtk-isp7x-controls.rst
new file mode 100755
index 000000000000..0e8d41d339e7
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/mtk-isp7x-controls.rst
@@ -0,0 +1,199 @@
+==========================================
+MediaTek ISP Camsys User Space Interface
+==========================================
+
+Overview
+--------
+
+The MediaTek ISP camsys driver provides a set of custom V4L2 controls for user space
+applications to configure and query the ISP hardware. These controls allow
+fine-grained management of resource allocation, feature enablement, and
+hardware-specific settings.
+
+Typical Usage Flow
+------------------
+
+1. Open the video device node (e.g., ``/dev/video0``).
+2. Set required V4L2 controls using ``VIDIOC_S_CTRL`` or ``VIDIOC_S_EXT_CTRLS``.
+3. Set up video format and buffers (``VIDIOC_S_FMT``, ``VIDIOC_REQBUFS``, etc.).
+4. Start streaming with ``VIDIOC_STREAMON``.
+5. Queue and dequeue buffers for image processing (``VIDIOC_QBUF``, ``VIDIOC_DQBUF``).
+6. Stop streaming with ``VIDIOC_STREAMOFF``.
+
+Control List and Documentation
+-----------------------------
+
+Below is a list of custom V4L2 controls provided by the camsys driver, with detailed descriptions.
+
+.. list-table::
+ :header-rows: 1
+
+ * - Control Name
+ - Type
+ - Description
+ - Valid Values / Range
+ - When to Set
+
+ * - ``V4L2_CID_MTK_CAM_USED_ENGINE_LIMIT``
+ - Integer
+ - Limits the maximum number of RAW engines (hardware pipelines) that can be used for this stream.
+ - 1 ~ 2 (default: 2)
+ - Before streaming
+
+ * - ``V4L2_CID_MTK_CAM_BIN_LIMIT``
+ - Integer
+ - Sets the maximum binning ratio allowed for the stream.
+ - 0 ~ 0xFFF (default: 0)
+ - Before streaming
+
+ * - ``V4L2_CID_MTK_CAM_FRZ_LIMIT``
+ - Integer
+ - Sets the maximum resizer (FRZ) ratio allowed (as a percentage).
+ - 70 ~ 100 (default: 100)
+ - Before streaming
+
+ * - ``V4L2_CID_MTK_CAM_RESOURCE_PLAN_POLICY``
+ - Integer (Enum)
+ - Selects the resource planning policy for the stream.
+ - 0: Default, 1: Power-saving, 2: Performance, ... (0~10)
+ - Before streaming
+
+ * - ``V4L2_CID_MTK_CAM_USED_ENGINE``
+ - Integer
+ - Reports or sets the number of RAW engines actually used for this stream after negotiation.
+ - 1 ~ 2 (default: 2)
+ - After resource negotiation
+
+ * - ``V4L2_CID_MTK_CAM_BIN``
+ - Boolean
+ - Enables or disables binning for the stream.
+ - 0 (disable), 1 (enable, default)
+ - Before streaming
+
+ * - ``V4L2_CID_MTK_CAM_FRZ``
+ - Integer
+ - Sets the resizer (FRZ) ratio (as a percentage) for the stream.
+ - 70 ~ 100 (default: 100)
+ - Before streaming
+
+ * - ``V4L2_CID_MTK_CAM_USED_ENGINE_TRY``
+ - Integer
+ - Reports the number of RAW engines that would be used for the current "try" format negotiation.
+ - 1 ~ 2 (default: 2)
+ - After VIDIOC_TRY_FMT
+
+ * - ``V4L2_CID_MTK_CAM_BIN_TRY``
+ - Boolean
+ - Reports whether binning would be enabled for the current "try" format negotiation.
+ - 0, 1 (default: 1)
+ - After VIDIOC_TRY_FMT
+
+ * - ``V4L2_CID_MTK_CAM_FRZ_TRY``
+ - Integer
+ - Reports the resizer ratio that would be used for the current "try" format negotiation.
+ - 70 ~ 100 (default: 100)
+ - After VIDIOC_TRY_FMT
+
+ * - ``V4L2_CID_MTK_CAM_PIXEL_RATE``
+ - Integer64
+ - Sets the pixel rate (in pixels per second) of the sensor for resource calculation.
+ - 0 ~ 0xFFFFFFFF
+ - Before streaming
+
+ * - ``V4L2_CID_MTK_CAM_FEATURE``
+ - Bitmask (Integer64)
+ - Bitmask of enabled features for the stream (e.g., HDR, 3A, etc.). See driver header for details.
+ - Bitmask (0 ~ RAW_FUNCTION_END)
+ - Before streaming or per-frame
+
+ * - ``V4L2_CID_MTK_CAM_SYNC_ID``
+ - Integer64
+ - Frame synchronization ID for multi-sensor or multi-pipeline synchronization.
+ - -1 ~ 0x7FFFFFFF (default: -1)
+ - Before streaming or per-frame
+
+ * - ``V4L2_CID_MTK_CAM_RAW_PATH_SELECT``
+ - Integer
+ - Selects the RAW path (e.g., main or secondary) for the stream.
+ - 0, 1 (default: 1)
+ - Before streaming
+
+ * - ``V4L2_CID_MTK_CAM_HSF_EN``
+ - Integer
+ - Enables or disables hardware secure flow (HSF) for the stream.
+ - 0 (disable), 1 (enable)
+ - Before streaming
+
+ * - ``V4L2_CID_MTK_CAM_PDE_INFO``
+ - Integer (struct mtk_cam_pde_info)
+ - Provides or sets phase detection (PDE) information for PDAF/AF features.
+ - See struct definition
+ - Before streaming or per-frame
+
+ * - ``V4L2_CID_MTK_CAM_MSTREAM_EXPOSURE``
+ - Integer/Integer64
+ - Controls multi-stream exposure settings for advanced HDR or multi-exposure modes.
+ - See driver header
+ - Before streaming or per-frame
+
+ * - ``V4L2_CID_MTK_CAM_RAW_RESOURCE_CALC``
+ - Compound (struct mtk_cam_resource)
+ - Triggers resource calculation for the current format and settings.
+ - See struct definition
+ - During format negotiation
+
+ * - ``V4L2_CID_MTK_CAM_TG_FLASH_CFG``
+ - Integer/struct
+ - Configures timing generator (TG) and flash synchronization.
+ - See driver header
+ - Before streaming or per-frame
+
+ * - ``V4L2_CID_MTK_CAM_RAW_RESOURCE_UPDATE``
+ - Integer
+ - Notifies the driver of a sensor mode update (e.g., seamless switch).
+ - 0 ~ 0xF (default: 0)
+ - When sensor mode changes
+
+ * - ``V4L2_CID_MTK_CAM_CAMSYS_HW_MODE``
+ - Integer64
+ - Selects the hardware mode for the CAMSYS pipeline (e.g., normal, secure, etc.).
+ - 0 ~ 0x7FFFFFFF
+ - Before streaming
+
+Control Usage Patterns
+---------------------
+
+- Controls with ``_TRY`` suffix are used to query the result of a "try" format negotiation (``VIDIOC_TRY_FMT``), without committing changes.
+- Controls without ``_TRY`` are used to set or get the actual configuration (``VIDIOC_S_FMT``, ``VIDIOC_G_CTRL``).
+- Use ``V4L2_CID_MTK_CAM_RAW_RESOURCE_CALC`` to check if the requested format and settings are feasible, and to get recommended values.
+- Use ``V4L2_CID_MTK_CAM_RAW_RESOURCE_UPDATE`` to notify the driver of a sensor mode change (e.g., seamless switch).
+- Use ``V4L2_CID_MTK_CAM_SYNC_ID`` to synchronize frames across multiple pipelines.
+- Use ``V4L2_CID_MTK_CAM_FEATURE`` to enable/disable advanced features (bitmask).
+- Use ``V4L2_CID_MTK_CAM_PDE_INFO`` to set/get phase detection information for PDAF/AF.
+- Use ``V4L2_CID_MTK_CAM_PIXEL_RATE`` to inform the driver of the sensor's pixel rate for bandwidth/resource calculation.
+
+Example: Setting a Control
+--------------------------
+
+.. code-block:: c
+
+ int fd = open("/dev/video0", O_RDWR);
+ struct v4l2_control ctrl = {
+ .id = V4L2_CID_MTK_CAM_BIN_LIMIT,
+ .value = 2,
+ };
+ if (ioctl(fd, VIDIOC_S_CTRL, &ctrl) < 0) {
+ perror("Failed to set BIN_LIMIT");
+ }
+
+Notes
+-----
+
+- All controls should be set before starting the video stream unless otherwise specified.
+- Refer to the driver header file for detailed struct definitions and bitmask values.
+- The driver will reject invalid values or unsupported configurations with -EINVAL.
+
+References
+----------
+
+- V4L2 documentation: https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/v4l2.html
\ No newline at end of file
--
2.18.0
More information about the dri-devel
mailing list